[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:34 dist-test-slave-xf24 krb5kdc[862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:34 dist-test-slave-xf24 krb5kdc[862](info): set up 2 sockets
Sep 26 04:07:34 dist-test-slave-xf24 krb5kdc[862](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 04:07:35.170019 855 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:35.296519 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34131
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:34131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:35.381709 875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:35.381939 875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:35.381963 875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:35.383176 875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:35.383229 875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:35.383249 875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:35.383265 875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:35.383281 875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:35.384915 875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:34131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:35.385221 875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:35.385499 875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:35.391237 883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.391232 880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:35.391306 875 server_base.cc:1034] running on GCE node
W20260926 04:07:35.391238 881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:35.391819 875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:35.392218 875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:35.393386 875 hybrid_clock.cc:648] HybridClock initialized: now 1790395655393365 us; error 39 us; skew 500 ppm
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:35.396227 875 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:35.396670 875 webserver.cc:460] Webserver started at http://127.0.213.254:39501/ using document root <none> and password file <none>
I20260926 04:07:35.396924 875 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:35.396996 875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:35.397159 875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:35.398273 875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "f6b31582c48d46998ce0492c9c14e908"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "f9728c9dc5298cc66349174f5f287449"
server_key_iv: "05bf31e771948f752629ae2071cea9be"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.398682 875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "f6b31582c48d46998ce0492c9c14e908"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "f9728c9dc5298cc66349174f5f287449"
server_key_iv: "05bf31e771948f752629ae2071cea9be"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.400355 875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 04:07:35.401288 890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.401522 875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:35.401643 875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "f6b31582c48d46998ce0492c9c14e908"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "f9728c9dc5298cc66349174f5f287449"
server_key_iv: "05bf31e771948f752629ae2071cea9be"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.401738 875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:35.418705 875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:35.421276 875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:35.421412 875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:35.425746 875 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:34131
I20260926 04:07:35.425798 942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:34131 every 8 connection(s)
I20260926 04:07:35.426374 875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:35.427174 943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:35.429658 943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908: Bootstrap starting.
I20260926 04:07:35.430286 943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:35.430708 943 log.cc:824] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:35.431581 943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908: No bootstrap required, opened a new log
I20260926 04:07:35.431591 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 875
I20260926 04:07:35.431721 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:35.432564 855 external_mini_cluster.cc:1428] Setting key d358a6b7ef03a6ec49633d6575025e63
I20260926 04:07:35.434191 943 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b31582c48d46998ce0492c9c14e908" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34131 } }
I20260926 04:07:35.434381 943 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:35.434434 943 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b31582c48d46998ce0492c9c14e908, State: Initialized, Role: FOLLOWER
I20260926 04:07:35.434604 943 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b31582c48d46998ce0492c9c14e908" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34131 } }
I20260926 04:07:35.434701 943 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:35.434741 943 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:35.434793 943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:35.435767 943 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b31582c48d46998ce0492c9c14e908" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34131 } }
I20260926 04:07:35.435894 943 leader_election.cc:302] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6b31582c48d46998ce0492c9c14e908; no voters:
I20260926 04:07:35.436113 948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:35.436112 943 leader_election.cc:288] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:35.436283 948 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [term 1 LEADER]: Becoming Leader. State: Replica: f6b31582c48d46998ce0492c9c14e908, State: Running, Role: LEADER
I20260926 04:07:35.436370 948 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b31582c48d46998ce0492c9c14e908" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34131 } }
I20260926 04:07:35.436446 943 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:35.436721 948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6b31582c48d46998ce0492c9c14e908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b31582c48d46998ce0492c9c14e908" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34131 } } }
I20260926 04:07:35.436731 950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b31582c48d46998ce0492c9c14e908. Latest consensus state: current_term: 1 leader_uuid: "f6b31582c48d46998ce0492c9c14e908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6b31582c48d46998ce0492c9c14e908" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34131 } } }
I20260926 04:07:35.436800 948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:35.436853 950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:35.437259 953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:35.437835 953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:35.439991 953 catalog_manager.cc:1347] Generated new cluster ID: 0da8e2a17a014ce6a39101494eca2e98
I20260926 04:07:35.440081 953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:35.443051 953 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:35.443571 953 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:35.451072 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.433625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.0.213.254:34131)
0926 04:07:35.433837 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:35.433848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:35.434110 (+ 262us) server_negotiation.cc:407] Connection header received
0926 04:07:35.436039 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:35.436047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:35.436133 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:35.436280 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:35.438916 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.440329 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.441402 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.441832 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.441957 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.446595 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:35.446601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:35.446601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:35.446616 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:35.448873 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.449470 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.449474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.449475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.449593 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.449885 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.449886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.449886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.450210 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:35.450365 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:35.450685 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 04:07:35.450847 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260926 04:07:35.451603 953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b31582c48d46998ce0492c9c14e908: Generated new TSK 0
I20260926 04:07:35.451774 953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:35.483729 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34131
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:35.569413 969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:35.569625 969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:35.569651 969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:35.570842 969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:35.570888 969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:35.570916 969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:35.572467 969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34131
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:35.572712 969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:35.572961 969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:35.578019 975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.578022 977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.578138 974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:35.578284 969 server_base.cc:1034] running on GCE node
I20260926 04:07:35.578572 969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:35.578775 969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:35.579964 969 hybrid_clock.cc:648] HybridClock initialized: now 1790395655579947 us; error 27 us; skew 500 ppm
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:35.582549 969 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:35.582988 969 webserver.cc:460] Webserver started at http://127.0.213.193:34505/ using document root <none> and password file <none>
I20260926 04:07:35.583256 969 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:35.583324 969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:35.583477 969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:35.584586 969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "88e2e5d575ca48b3a482aca2274f7cb0"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "103143d3831fa62671e9b9129fe0f251"
server_key_iv: "2451288aa8050679eebc06ba39961d0f"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.584952 969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "88e2e5d575ca48b3a482aca2274f7cb0"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "103143d3831fa62671e9b9129fe0f251"
server_key_iv: "2451288aa8050679eebc06ba39961d0f"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.586256 969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:35.587105 984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.587309 969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:35.587412 969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "88e2e5d575ca48b3a482aca2274f7cb0"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "103143d3831fa62671e9b9129fe0f251"
server_key_iv: "2451288aa8050679eebc06ba39961d0f"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.587488 969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:35.604967 969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:35.607088 969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:35.607262 969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:35.607510 969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:35.607824 969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:35.607878 969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.607913 969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:35.607970 969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.614580 969 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:42057
I20260926 04:07:35.614617 1097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:42057 every 8 connection(s)
I20260926 04:07:35.614951 969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:35.618706 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 969
I20260926 04:07:35.618800 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:35.618952 855 external_mini_cluster.cc:1428] Setting key 3a1b69f9a9358c0c5bc39338b5cad87b
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:35.627033 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.615782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:56535 (local address 127.0.213.254:34131)
0926 04:07:35.615848 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:35.615850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:35.616049 (+ 199us) server_negotiation.cc:407] Connection header received
0926 04:07:35.616619 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:35.616620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:35.616631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:35.616700 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:35.617767 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.618284 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.619717 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.619903 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.619959 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.622603 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:35.622605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:35.622605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:35.622615 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:35.625169 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.625882 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.625884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.625885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.625969 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.626331 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.626333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.626333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.626471 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:35.626554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:35.626842 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 04:07:35.626935 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:07:35.627267 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.615738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:34131 (local address 127.0.213.193:56535)
0926 04:07:35.615908 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:35.615951 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:35.616494 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:35.616828 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:35.616832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:35.617451 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:35.617651 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:35.617656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.618370 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.618371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:35.619641 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:35.619644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.619987 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.619988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:35.620183 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.620791 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:35.620802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:35.622422 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:35.625317 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:35.625328 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:35.625331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:35.625764 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:35.626061 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:35.626062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:35.626062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:35.626258 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:35.626544 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:35.626550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:35.626705 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 04:07:35.626822 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:07:35.626994 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260926 04:07:35.628042 1098 heartbeater.cc:347] Connected to a master server at 127.0.213.254:34131
I20260926 04:07:35.628219 1098 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:35.628381 1098 heartbeater.cc:511] Master 127.0.213.254:34131 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:07:35.628938 907 ts_manager.cc:194] Registered new tserver with Master: 88e2e5d575ca48b3a482aca2274f7cb0 (127.0.213.193:42057)
I20260926 04:07:35.629575 907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:56535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:35.647357 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34131
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:35.736182 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:35.736451 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:35.736493 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:35.737880 1103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:35.737963 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:35.738023 1103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:35.739622 1103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34131
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:35.739905 1103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:35.740216 1103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:35.746541 1108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.746502 1109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.746479 1111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:35.746644 1103 server_base.cc:1034] running on GCE node
I20260926 04:07:35.747144 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:35.747468 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:35.748673 1103 hybrid_clock.cc:648] HybridClock initialized: now 1790395655748616 us; error 72 us; skew 500 ppm
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:35.751827 1103 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:35.752429 1103 webserver.cc:460] Webserver started at http://127.0.213.194:38051/ using document root <none> and password file <none>
I20260926 04:07:35.752704 1103 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:35.752776 1103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:35.752938 1103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:35.754362 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "1f0f4c0828a44f0cb8b3eb7da7724331"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "b235b0631ede64143eb3f9b9d4f31eb1"
server_key_iv: "4a0c7a9b08016c3a2326d1a32f991d8b"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.754922 1103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "1f0f4c0828a44f0cb8b3eb7da7724331"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "b235b0631ede64143eb3f9b9d4f31eb1"
server_key_iv: "4a0c7a9b08016c3a2326d1a32f991d8b"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.756624 1103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260926 04:07:35.757889 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.758234 1103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:35.758373 1103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "1f0f4c0828a44f0cb8b3eb7da7724331"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "b235b0631ede64143eb3f9b9d4f31eb1"
server_key_iv: "4a0c7a9b08016c3a2326d1a32f991d8b"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.758468 1103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:35.777782 1103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:35.780661 1103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:35.780857 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:35.781121 1103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:35.781451 1103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:35.781517 1103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.781563 1103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:35.781591 1103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.789024 1103 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:45381
I20260926 04:07:35.789080 1231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:45381 every 8 connection(s)
I20260926 04:07:35.789640 1103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:35.793324 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1103
I20260926 04:07:35.793447 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:35.793633 855 external_mini_cluster.cc:1428] Setting key 981f9a4934f44e3e1499d393fed9349b
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:35.800987 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.790481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:38331 (local address 127.0.213.254:34131)
0926 04:07:35.790552 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:35.790554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:35.790846 (+ 292us) server_negotiation.cc:407] Connection header received
0926 04:07:35.791366 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:35.791367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:35.791379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:35.791450 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:35.792537 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.793034 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.794245 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.794529 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.794608 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.797573 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:35.797575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:35.797575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:35.797583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:35.799242 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.799844 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.799846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.799846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.799913 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.800260 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.800261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.800261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.800396 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:35.800454 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:35.800833 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 04:07:35.800915 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:07:35.801260 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.790472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:34131 (local address 127.0.213.194:38331)
0926 04:07:35.790691 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:35.790737 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:35.791248 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:35.791518 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:35.791522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:35.792204 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:35.792423 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:35.792429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.793181 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.793182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:35.794171 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:35.794173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.794652 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.794653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:35.794783 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.795402 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:35.795412 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:35.797438 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:35.799345 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:35.799354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:35.799357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:35.799738 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:35.799998 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:35.799999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:35.799999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:35.800186 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:35.800475 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:35.800482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:35.800672 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 04:07:35.800794 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 04:07:35.800990 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20260926 04:07:35.801815 1232 heartbeater.cc:347] Connected to a master server at 127.0.213.254:34131
I20260926 04:07:35.801931 1232 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:35.802150 1232 heartbeater.cc:511] Master 127.0.213.254:34131 requested a full tablet report, sending...
I20260926 04:07:35.802558 907 ts_manager.cc:194] Registered new tserver with Master: 1f0f4c0828a44f0cb8b3eb7da7724331 (127.0.213.194:45381)
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:07:35.802991 907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:38331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:35.824774 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34131
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:35.909849 1237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:35.910090 1237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:35.910116 1237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:35.911258 1237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:35.911294 1237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:35.911322 1237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:35.912858 1237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34131
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:35.913097 1237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:35.913290 1237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:35.918462 1242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.918452 1245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:35.918457 1243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:35.918678 1237 server_base.cc:1034] running on GCE node
I20260926 04:07:35.918946 1237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:35.919168 1237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:35.920305 1237 hybrid_clock.cc:648] HybridClock initialized: now 1790395655920291 us; error 34 us; skew 500 ppm
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:35.922922 1237 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:35.923424 1237 webserver.cc:460] Webserver started at http://127.0.213.195:33695/ using document root <none> and password file <none>
I20260926 04:07:35.923657 1237 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:35.923727 1237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:35.923911 1237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:35.925084 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "503f97a0ba684ef68381dffab383940a"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "4e1392cdae0ba7759a90df6b53529648"
server_key_iv: "3ed910b037dc4e30a867ec1a7165fb35"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.925457 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "503f97a0ba684ef68381dffab383940a"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "4e1392cdae0ba7759a90df6b53529648"
server_key_iv: "3ed910b037dc4e30a867ec1a7165fb35"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.926802 1237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:07:35.927592 1252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.927783 1237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:35.927879 1237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "503f97a0ba684ef68381dffab383940a"
format_stamp: "Formatted at 2026-09-26 04:07:35 on dist-test-slave-xf24"
server_key: "4e1392cdae0ba7759a90df6b53529648"
server_key_iv: "3ed910b037dc4e30a867ec1a7165fb35"
server_key_version: "encryptionkey@0"
I20260926 04:07:35.927961 1237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:35.943382 1237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:35.945772 1237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:35.945981 1237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:35.946259 1237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:35.946600 1237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:35.946658 1237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.946704 1237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:35.946730 1237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:35.953389 1237 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:46317
I20260926 04:07:35.953456 1365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:46317 every 8 connection(s)
I20260926 04:07:35.953795 1237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:35.959949 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1237
I20260926 04:07:35.960058 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:35.960274 855 external_mini_cluster.cc:1428] Setting key 6439b8e784218d5fb0baf5417978bc62
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:35.964280 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.954585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:55795 (local address 127.0.213.254:34131)
0926 04:07:35.954630 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:35.954632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:35.954846 (+ 214us) server_negotiation.cc:407] Connection header received
0926 04:07:35.955350 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:35.955352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:35.955362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:35.955434 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:35.956456 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.956988 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.958165 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.958363 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.958424 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.960866 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:35.960868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:35.960868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:35.960877 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:35.962658 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.963064 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.963065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.963065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.963116 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.963470 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.963471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.963471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.963595 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:35.963661 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:35.964108 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 04:07:35.964204 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:07:35.964531 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.954572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:34131 (local address 127.0.213.195:55795)
0926 04:07:35.954721 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:35.954756 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:35.955262 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:35.955499 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:35.955503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:35.956144 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:35.956345 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:35.956350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.957102 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.957103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:35.958080 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:35.958083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.958443 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.958444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:35.958559 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.959085 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:35.959093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:35.960779 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:35.962734 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:35.962740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:35.962742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:35.962995 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:35.963179 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:35.963180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:35.963180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:35.963363 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:35.963699 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:35.963705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:35.963893 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 04:07:35.964007 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 04:07:35.964242 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260926 04:07:35.965147 1366 heartbeater.cc:347] Connected to a master server at 127.0.213.254:34131
I20260926 04:07:35.965245 1366 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:35.965469 1366 heartbeater.cc:511] Master 127.0.213.254:34131 requested a full tablet report, sending...
I20260926 04:07:35.965940 907 ts_manager.cc:194] Registered new tserver with Master: 503f97a0ba684ef68381dffab383940a (127.0.213.195:46317)
I20260926 04:07:35.966331 907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:55795
I20260926 04:07:35.972797 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 04:07:35 dist-test-slave-xf24 krb5kdc[862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:35.997335 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:35.988629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.0.213.193:42057)
0926 04:07:35.988822 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:35.988827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:35.988855 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:07:35.988880 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:35.988882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:35.988896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:35.988960 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:35.989763 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.990634 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.991559 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:35.991935 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:35.992005 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:35.994447 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:35.994461 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:35.994461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:35.994468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:35.996199 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.996564 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.996567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.996568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.996626 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:35.996805 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:35.996806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:35.996807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:35.996977 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:35.997103 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:35.997159 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 04:07:35.997219 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
W20260926 04:07:35.997769 1003 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46684
Sep 26 04:07:36 dist-test-slave-xf24 krb5kdc[862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 04:07:36 dist-test-slave-xf24 krb5kdc[862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:36.022573 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:36.014753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.0.213.193:42057)
0926 04:07:36.014827 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:36.014829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:36.014927 (+ 98us) server_negotiation.cc:407] Connection header received
0926 04:07:36.015027 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:36.015028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:36.015041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:36.015113 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:36.015758 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:36.016337 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:36.017095 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:36.017300 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:36.017375 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:36.019692 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:36.019693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:36.019694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:36.019708 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:36.021346 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:36.021751 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:36.021753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:36.021753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:36.021812 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:36.022015 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:36.022016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:36.022023 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:36.022133 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:36.022268 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:36.022358 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 04:07:36.022458 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:07:36.023217 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 969
I20260926 04:07:36.028142 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1103
I20260926 04:07:36.033780 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1237
I20260926 04:07:36.038182 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 875
2026-09-26T04:07:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1926 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:36 dist-test-slave-xf24 krb5kdc[1380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:36 dist-test-slave-xf24 krb5kdc[1380](info): set up 2 sockets
Sep 26 04:07:36 dist-test-slave-xf24 krb5kdc[1380](info): commencing operation
krb5kdc: starting...
W20260926 04:07:37.083877 855 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:37.211143 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40589
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:37.296702 1393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:37.296943 1393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:37.296968 1393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:37.298328 1393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:37.298398 1393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:37.298413 1393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:37.298427 1393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:37.298448 1393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:37.300041 1393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:37.300302 1393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:37.300495 1393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:37.305809 1399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.305836 1398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:37.305868 1393 server_base.cc:1034] running on GCE node
W20260926 04:07:37.305830 1401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:37.306363 1393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:37.306635 1393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:37.307785 1393 hybrid_clock.cc:648] HybridClock initialized: now 1790395657307754 us; error 51 us; skew 500 ppm
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:37.310705 1393 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:37.311233 1393 webserver.cc:460] Webserver started at http://127.0.213.254:42503/ using document root <none> and password file <none>
I20260926 04:07:37.311496 1393 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:37.311569 1393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:37.311730 1393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:37.312951 1393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "d76d8d3486334292863f619a73b0be76"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "3de16a957c9fc999fd32f4668c08c2a9"
server_key_iv: "c9f561e3b6b46c09c0077ab033071501"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.313340 1393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "d76d8d3486334292863f619a73b0be76"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "3de16a957c9fc999fd32f4668c08c2a9"
server_key_iv: "c9f561e3b6b46c09c0077ab033071501"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.314774 1393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:07:37.315703 1408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.316051 1393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:37.316190 1393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "d76d8d3486334292863f619a73b0be76"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "3de16a957c9fc999fd32f4668c08c2a9"
server_key_iv: "c9f561e3b6b46c09c0077ab033071501"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.316278 1393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:37.330467 1393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:37.333011 1393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:37.333217 1393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:37.337436 1393 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:40589
I20260926 04:07:37.337476 1460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:40589 every 8 connection(s)
I20260926 04:07:37.337836 1393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:37.338714 1461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:37.341096 1461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76: Bootstrap starting.
I20260926 04:07:37.341738 1461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:37.342054 1461 log.cc:824] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:37.342808 1461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76: No bootstrap required, opened a new log
I20260926 04:07:37.344475 1461 raft_consensus.cc:348] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d8d3486334292863f619a73b0be76" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40589 } }
I20260926 04:07:37.344624 1461 raft_consensus.cc:374] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:37.344647 1461 raft_consensus.cc:729] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76d8d3486334292863f619a73b0be76, State: Initialized, Role: FOLLOWER
I20260926 04:07:37.344806 1461 consensus_queue.cc:260] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d8d3486334292863f619a73b0be76" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40589 } }
I20260926 04:07:37.344892 1461 raft_consensus.cc:388] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:37.344916 1461 raft_consensus.cc:482] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:37.344985 1461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:37.345542 1461 raft_consensus.cc:504] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d8d3486334292863f619a73b0be76" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40589 } }
I20260926 04:07:37.345656 1461 leader_election.cc:302] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d76d8d3486334292863f619a73b0be76; no voters:
I20260926 04:07:37.345865 1461 leader_election.cc:288] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:37.345940 1464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:37.346168 1461 sys_catalog.cc:564] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:37.346377 1464 raft_consensus.cc:686] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [term 1 LEADER]: Becoming Leader. State: Replica: d76d8d3486334292863f619a73b0be76, State: Running, Role: LEADER
I20260926 04:07:37.346607 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1393
I20260926 04:07:37.346705 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:37.346879 855 external_mini_cluster.cc:1428] Setting key 17cb40bf56b5e3b3d718de4ca622e883
I20260926 04:07:37.347137 1464 consensus_queue.cc:237] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d8d3486334292863f619a73b0be76" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40589 } }
I20260926 04:07:37.347683 1466 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d76d8d3486334292863f619a73b0be76. Latest consensus state: current_term: 1 leader_uuid: "d76d8d3486334292863f619a73b0be76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d8d3486334292863f619a73b0be76" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40589 } } }
I20260926 04:07:37.347716 1465 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d76d8d3486334292863f619a73b0be76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d76d8d3486334292863f619a73b0be76" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40589 } } }
I20260926 04:07:37.347780 1466 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:37.347795 1465 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:37.348136 1479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:37.348673 1479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:37.351090 1479 catalog_manager.cc:1347] Generated new cluster ID: 26259ac3e2ea474aaf775e4be9133b3f
I20260926 04:07:37.351161 1479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:37.360174 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.348147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.0.213.254:40589)
0926 04:07:37.348702 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.348710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.348735 (+ 25us) server_negotiation.cc:407] Connection header received
0926 04:07:37.348888 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.348894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.348940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.349059 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.349775 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.350923 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.352010 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.352728 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.352858 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.355895 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.355900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.355901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.355913 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.357982 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.358412 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.358424 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.358424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.358581 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.358794 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.358796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.358796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.359318 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.359509 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.359591 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.359881 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":71,"threads_started":1}
I20260926 04:07:37.363005 1479 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:37.363714 1479 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:37.368412 1479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d76d8d3486334292863f619a73b0be76: Generated new TSK 0
I20260926 04:07:37.368605 1479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:37.400331 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40589
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:37.486536 1487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:37.486786 1487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:37.486814 1487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:37.488190 1487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:37.488242 1487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:37.488272 1487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:37.489833 1487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40589
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:37.490106 1487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:37.490366 1487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:37.495748 1492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.495750 1493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.495750 1495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:37.496039 1487 server_base.cc:1034] running on GCE node
I20260926 04:07:37.496320 1487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:37.496590 1487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:37.497767 1487 hybrid_clock.cc:648] HybridClock initialized: now 1790395657497755 us; error 41 us; skew 500 ppm
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:37.500625 1487 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:37.501091 1487 webserver.cc:460] Webserver started at http://127.0.213.193:41925/ using document root <none> and password file <none>
I20260926 04:07:37.501335 1487 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:37.501410 1487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:37.501552 1487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:37.502655 1487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "a07364af14c5458e95e31e44a1ffbb5b"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "69d20d20ff5eb1a7292a0dc1b4314e26"
server_key_iv: "0a6c421dab6eb01a702e6118274e6b00"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.503021 1487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "a07364af14c5458e95e31e44a1ffbb5b"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "69d20d20ff5eb1a7292a0dc1b4314e26"
server_key_iv: "0a6c421dab6eb01a702e6118274e6b00"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.504545 1487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:37.505528 1502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.505805 1487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:37.505928 1487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "a07364af14c5458e95e31e44a1ffbb5b"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "69d20d20ff5eb1a7292a0dc1b4314e26"
server_key_iv: "0a6c421dab6eb01a702e6118274e6b00"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.506017 1487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:37.521574 1487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:37.523653 1487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:37.523829 1487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:37.524143 1487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:37.524494 1487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:37.524557 1487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.524600 1487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:37.524646 1487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.530351 1487 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:45235
I20260926 04:07:37.530401 1615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:45235 every 8 connection(s)
I20260926 04:07:37.530732 1487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:37.535662 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1487
I20260926 04:07:37.535765 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:37.535941 855 external_mini_cluster.cc:1428] Setting key 43f8270ad5749b8d030027eb9e1b640c
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:37.541680 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.531547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:57235 (local address 127.0.213.254:40589)
0926 04:07:37.531637 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.531640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.531838 (+ 198us) server_negotiation.cc:407] Connection header received
0926 04:07:37.532559 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.532560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.532585 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.532657 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.533685 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.534262 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.535393 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.535798 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.535896 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.538388 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.538389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.538390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.538398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.539958 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.540460 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.540461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.540461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.540527 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.540807 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.540808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.540808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.540941 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.541116 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.541459 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.541568 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 04:07:37.541857 1618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.531515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40589 (local address 127.0.213.193:57235)
0926 04:07:37.531689 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.531734 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.532279 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.532762 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.532766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.533354 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:37.533564 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.533569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.534374 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.534375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.535230 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.535233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.535934 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.535935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.536155 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.536835 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:37.536844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:37.538283 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:37.540088 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:37.540097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:37.540100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:37.540349 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:37.540608 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:37.540609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:37.540610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:37.540732 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:37.541037 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:37.541093 (+ 56us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:37.541287 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.541413 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.541595 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260926 04:07:37.542832 1616 heartbeater.cc:347] Connected to a master server at 127.0.213.254:40589
I20260926 04:07:37.542968 1616 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:37.543195 1616 heartbeater.cc:511] Master 127.0.213.254:40589 requested a full tablet report, sending...
I20260926 04:07:37.543905 1425 ts_manager.cc:194] Registered new tserver with Master: a07364af14c5458e95e31e44a1ffbb5b (127.0.213.193:45235)
I20260926 04:07:37.544914 1425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:57235
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:37.567813 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40589
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:37.651661 1621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:37.651892 1621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:37.651921 1621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:37.653122 1621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:37.653164 1621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:37.653190 1621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:37.654611 1621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40589
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:37.654824 1621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:37.655005 1621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:37.660938 1626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.660972 1629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.660938 1627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:37.661209 1621 server_base.cc:1034] running on GCE node
I20260926 04:07:37.661448 1621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:37.661741 1621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:37.662926 1621 hybrid_clock.cc:648] HybridClock initialized: now 1790395657662912 us; error 35 us; skew 500 ppm
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:37.665850 1621 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:37.666361 1621 webserver.cc:460] Webserver started at http://127.0.213.194:37907/ using document root <none> and password file <none>
I20260926 04:07:37.666630 1621 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:37.666700 1621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:37.666862 1621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:37.668169 1621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "2d01b683935342d3add6a5186ce6e250"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "2b19355765a3a865cbe4bf9405cb2b8c"
server_key_iv: "ce7ccd5dcee292abd1c4579b5443b204"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.668558 1621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "2d01b683935342d3add6a5186ce6e250"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "2b19355765a3a865cbe4bf9405cb2b8c"
server_key_iv: "ce7ccd5dcee292abd1c4579b5443b204"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.670013 1621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:07:37.670913 1636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.671211 1621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:37.671325 1621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "2d01b683935342d3add6a5186ce6e250"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "2b19355765a3a865cbe4bf9405cb2b8c"
server_key_iv: "ce7ccd5dcee292abd1c4579b5443b204"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.671442 1621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:37.680146 1621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:37.682659 1621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:37.682832 1621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:37.683097 1621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:37.683434 1621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:37.683490 1621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.683534 1621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:37.683559 1621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.689612 1621 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:40159
I20260926 04:07:37.689685 1749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:40159 every 8 connection(s)
I20260926 04:07:37.690037 1621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:37.692911 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1621
I20260926 04:07:37.692988 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:37.693166 855 external_mini_cluster.cc:1428] Setting key 01331f7d4f89824fe1ce95be2fe101a6
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:37.700136 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.690805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:33047 (local address 127.0.213.254:40589)
0926 04:07:37.690855 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.690857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.691083 (+ 226us) server_negotiation.cc:407] Connection header received
0926 04:07:37.691626 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.691628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.691640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.691697 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.692753 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.693440 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.694536 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.694747 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.694823 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.697175 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.697177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.697177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.697186 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.698625 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.699092 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.699093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.699094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.699158 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.699423 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.699424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.699424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.699546 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.699625 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.699916 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.700011 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:07:37.700320 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.690784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40589 (local address 127.0.213.194:33047)
0926 04:07:37.690955 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.690992 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.691515 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.691788 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.691793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.692455 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:37.692651 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.692656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.693516 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.693517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.694439 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.694441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.694845 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.694846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.694976 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.695555 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:37.695564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:37.697076 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:37.698753 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:37.698761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:37.698763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:37.699010 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:37.699225 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:37.699226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:37.699226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:37.699348 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:37.699615 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:37.699621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:37.699772 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.699881 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.700044 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:07:37.700930 1750 heartbeater.cc:347] Connected to a master server at 127.0.213.254:40589
I20260926 04:07:37.701043 1750 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:37.701227 1750 heartbeater.cc:511] Master 127.0.213.254:40589 requested a full tablet report, sending...
I20260926 04:07:37.701678 1425 ts_manager.cc:194] Registered new tserver with Master: 2d01b683935342d3add6a5186ce6e250 (127.0.213.194:40159)
I20260926 04:07:37.702105 1425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:33047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:37.720266 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40589
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:37.803884 1755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:37.804194 1755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:37.804242 1755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:37.805418 1755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:37.805492 1755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:37.805522 1755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:37.806910 1755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40589
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:37.807111 1755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:37.807300 1755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:37.812474 1760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.812503 1761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:37.812493 1763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:37.812717 1755 server_base.cc:1034] running on GCE node
I20260926 04:07:37.812943 1755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:37.813194 1755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:37.814360 1755 hybrid_clock.cc:648] HybridClock initialized: now 1790395657814347 us; error 32 us; skew 500 ppm
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:37.816962 1755 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:37.817446 1755 webserver.cc:460] Webserver started at http://127.0.213.195:41719/ using document root <none> and password file <none>
I20260926 04:07:37.817677 1755 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:37.817744 1755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:37.817894 1755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:37.818971 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "79effd792dff42f6b6edfa3b92f29d05"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "2cd8d5f36bc8ba7faa545a68613ae71e"
server_key_iv: "3d651a09bee2f179fd29e7b67a0f1d09"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.819370 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "79effd792dff42f6b6edfa3b92f29d05"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "2cd8d5f36bc8ba7faa545a68613ae71e"
server_key_iv: "3d651a09bee2f179fd29e7b67a0f1d09"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.820832 1755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 04:07:37.821671 1770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.821907 1755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:37.822021 1755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "79effd792dff42f6b6edfa3b92f29d05"
format_stamp: "Formatted at 2026-09-26 04:07:37 on dist-test-slave-xf24"
server_key: "2cd8d5f36bc8ba7faa545a68613ae71e"
server_key_iv: "3d651a09bee2f179fd29e7b67a0f1d09"
server_key_version: "encryptionkey@0"
I20260926 04:07:37.822099 1755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:37.855293 1755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:37.857605 1755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:37.857766 1755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:37.858016 1755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:37.858316 1755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:37.858369 1755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.858410 1755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:37.858434 1755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:37.864806 1755 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:45681
I20260926 04:07:37.864847 1883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:45681 every 8 connection(s)
I20260926 04:07:37.865186 1755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:37.865891 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1755
I20260926 04:07:37.866020 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:37.866180 855 external_mini_cluster.cc:1428] Setting key 06f2ffd941e29055807e70424b10cd34
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:37.875010 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.865930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:49249 (local address 127.0.213.254:40589)
0926 04:07:37.866006 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.866008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.866182 (+ 174us) server_negotiation.cc:407] Connection header received
0926 04:07:37.866727 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.866727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.866739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.866808 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.867826 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.868482 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.869495 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.869688 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.869754 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.871996 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.871997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.871998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.872004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.873512 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.873915 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.873916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.873916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.873983 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.874234 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.874235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.874235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.874367 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.874433 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.874753 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.874943 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:07:37.875074 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.865892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40589 (local address 127.0.213.195:49249)
0926 04:07:37.866042 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.866085 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.866608 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.866876 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.866880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.867521 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:37.867722 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.867727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.868574 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.868575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.869411 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.869413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.869792 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.869793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.869916 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.870438 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:37.870447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:37.871913 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:37.873589 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:37.873594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:37.873596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:37.873823 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:37.874054 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:37.874054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:37.874055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:37.874158 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:37.874437 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:37.874441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:37.874561 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.874687 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.874882 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260926 04:07:37.875629 1884 heartbeater.cc:347] Connected to a master server at 127.0.213.254:40589
I20260926 04:07:37.875721 1884 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:37.875864 1884 heartbeater.cc:511] Master 127.0.213.254:40589 requested a full tablet report, sending...
I20260926 04:07:37.876303 1425 ts_manager.cc:194] Registered new tserver with Master: 79effd792dff42f6b6edfa3b92f29d05 (127.0.213.195:45681)
I20260926 04:07:37.876681 1425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:49249
I20260926 04:07:37.877919 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:07:37.885732 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.879372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.213.254:40589)
0926 04:07:37.879419 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.879421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.879489 (+ 68us) server_negotiation.cc:407] Connection header received
0926 04:07:37.879564 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.879565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.879574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.879648 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.880240 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.880718 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.881467 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.881656 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.881741 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.882986 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.882987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.882987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.882994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.884533 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.884914 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.884916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.884916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.884975 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.885180 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.885181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.885181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.885337 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.885446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.885552 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.885630 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:07:37.888121 1425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:37.889585 1425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:37.896328 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.892243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:40159 (local address 127.0.0.1:42282)
0926 04:07:37.892611 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.892620 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.892672 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.892938 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.892939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.892942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:37.893019 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.893021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.893927 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.893928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.895103 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.895105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.895902 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.895903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.896101 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.896122 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.896190 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.896247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":15,"thread_start_us":54,"threads_started":1}
I20260926 04:07:37.896353 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.892365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.0.213.194:40159)
0926 04:07:37.892571 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.892576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.892677 (+ 101us) server_negotiation.cc:407] Connection header received
0926 04:07:37.892755 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.892757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.892774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.892853 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:37.893084 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.893822 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.895188 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.895804 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.895890 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.895944 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.896232 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.896270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":139,"threads_started":1}
I20260926 04:07:37.896572 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.891994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:45681 (local address 127.0.0.1:46810)
0926 04:07:37.892143 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.892184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.892280 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.892965 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.892966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.892967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:37.892999 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.893000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.894396 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.894397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.895457 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.895458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.896369 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.896370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.896430 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.896434 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.896465 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.896501 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":49,"threads_started":1}
I20260926 04:07:37.896893 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.892143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.0.213.195:45681)
0926 04:07:37.892798 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.892802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.892811 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:37.892847 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.892849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.892864 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.893067 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:37.893138 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.894263 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.895540 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.896259 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.896688 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.896750 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.896776 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.896816 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":99,"threads_started":1}
I20260926 04:07:37.897363 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.892000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:45235 (local address 127.0.0.1:40210)
0926 04:07:37.892469 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.892481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.892534 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.892897 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.892900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.892903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:37.892979 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.892981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.895215 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.895215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.896112 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.896114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.897021 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.897022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.897094 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.897098 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.897129 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.897168 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":98,"threads_started":1}
I20260926 04:07:37.897491 1818 tablet_service.cc:1467] Processing CreateTablet for tablet 602edbdf44524e0ab04eb60017f36814 (DEFAULT_TABLE table=test-table [id=8eca22f4e04c4927a215a267ab77c646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:37.897493 1684 tablet_service.cc:1467] Processing CreateTablet for tablet 602edbdf44524e0ab04eb60017f36814 (DEFAULT_TABLE table=test-table [id=8eca22f4e04c4927a215a267ab77c646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:37.897670 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.892462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.213.193:45235)
0926 04:07:37.892655 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.892660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.892670 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:37.892738 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.892741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.892759 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.893061 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:37.893133 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.894262 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.896198 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.896917 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.897516 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.897562 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.897582 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.897612 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260926 04:07:37.897836 1818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 602edbdf44524e0ab04eb60017f36814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:37.897838 1684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 602edbdf44524e0ab04eb60017f36814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:37.898228 1550 tablet_service.cc:1467] Processing CreateTablet for tablet 602edbdf44524e0ab04eb60017f36814 (DEFAULT_TABLE table=test-table [id=8eca22f4e04c4927a215a267ab77c646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:37.898473 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 602edbdf44524e0ab04eb60017f36814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:37.901000 1901 tablet_bootstrap.cc:492] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b: Bootstrap starting.
I20260926 04:07:37.901366 1902 tablet_bootstrap.cc:492] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: Bootstrap starting.
I20260926 04:07:37.901664 1901 tablet_bootstrap.cc:654] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:37.901943 1903 tablet_bootstrap.cc:492] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05: Bootstrap starting.
I20260926 04:07:37.901995 1902 tablet_bootstrap.cc:654] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:37.902339 1901 log.cc:824] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:37.902585 1902 log.cc:824] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:37.902648 1903 tablet_bootstrap.cc:654] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:37.903101 1901 tablet_bootstrap.cc:492] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b: No bootstrap required, opened a new log
I20260926 04:07:37.903128 1903 log.cc:824] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:37.903189 1901 ts_tablet_manager.cc:1397] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:37.903927 1903 tablet_bootstrap.cc:492] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05: No bootstrap required, opened a new log
I20260926 04:07:37.904000 1903 ts_tablet_manager.cc:1397] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:37.904362 1902 tablet_bootstrap.cc:492] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: No bootstrap required, opened a new log
I20260926 04:07:37.904449 1902 ts_tablet_manager.cc:1397] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:07:37.905155 1901 raft_consensus.cc:348] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.905351 1901 raft_consensus.cc:374] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:37.905395 1901 raft_consensus.cc:729] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07364af14c5458e95e31e44a1ffbb5b, State: Initialized, Role: FOLLOWER
I20260926 04:07:37.905512 1901 consensus_queue.cc:260] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.905784 1901 ts_tablet_manager.cc:1428] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:37.905812 1616 heartbeater.cc:503] Master 127.0.213.254:40589 was elected leader, sending a full tablet report...
I20260926 04:07:37.906018 1903 raft_consensus.cc:348] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.906199 1903 raft_consensus.cc:374] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:37.906077 1902 raft_consensus.cc:348] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.906235 1903 raft_consensus.cc:729] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79effd792dff42f6b6edfa3b92f29d05, State: Initialized, Role: FOLLOWER
I20260926 04:07:37.906244 1902 raft_consensus.cc:374] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:37.906284 1902 raft_consensus.cc:729] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d01b683935342d3add6a5186ce6e250, State: Initialized, Role: FOLLOWER
I20260926 04:07:37.906394 1902 consensus_queue.cc:260] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.906561 1903 consensus_queue.cc:260] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.906666 1750 heartbeater.cc:503] Master 127.0.213.254:40589 was elected leader, sending a full tablet report...
I20260926 04:07:37.906669 1902 ts_tablet_manager.cc:1428] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:37.906916 1903 ts_tablet_manager.cc:1428] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:37.906930 1884 heartbeater.cc:503] Master 127.0.213.254:40589 was elected leader, sending a full tablet report...
I20260926 04:07:37.918658 1909 raft_consensus.cc:482] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:37.918856 1909 raft_consensus.cc:504] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.919189 1909 leader_election.cc:288] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07364af14c5458e95e31e44a1ffbb5b (127.0.213.193:45235), 2d01b683935342d3add6a5186ce6e250 (127.0.213.194:40159)
I20260926 04:07:37.922267 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.919376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:45235 (local address 127.0.213.195:54801)
0926 04:07:37.919436 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.919449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.919524 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.919732 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.919734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.919735 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:37.919808 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.919810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.920552 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.920553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.921367 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.921370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.921998 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.921999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.922080 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.922083 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.922142 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.922174 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:07:37.922272 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.919488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:54801 (local address 127.0.213.193:45235)
0926 04:07:37.919538 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.919541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.919551 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:37.919583 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.919584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.919595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.919652 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:37.919910 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.920474 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.921498 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.921920 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.921977 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.921984 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.922156 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.922198 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 04:07:37.922366 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.919671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:38443 (local address 127.0.213.194:40159)
0926 04:07:37.919751 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.919753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.919762 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:37.919808 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.919809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.919821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.919880 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:37.920034 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.920601 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.921595 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.922000 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.922079 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.922087 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.922249 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.922287 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 04:07:37.922410 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.919485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:40159 (local address 127.0.213.195:38443)
0926 04:07:37.919707 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:37.919716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:37.919760 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:37.919902 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:37.919903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:37.919904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:37.919969 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.919970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.920692 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.920693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.921511 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:37.921513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.922089 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.922090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:37.922171 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.922175 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:37.922231 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:07:37.922266 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":136,"threads_started":1}
I20260926 04:07:37.922721 1704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602edbdf44524e0ab04eb60017f36814" candidate_uuid: "79effd792dff42f6b6edfa3b92f29d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d01b683935342d3add6a5186ce6e250" is_pre_election: true
I20260926 04:07:37.922721 1570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602edbdf44524e0ab04eb60017f36814" candidate_uuid: "79effd792dff42f6b6edfa3b92f29d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07364af14c5458e95e31e44a1ffbb5b" is_pre_election: true
I20260926 04:07:37.922878 1704 raft_consensus.cc:2454] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79effd792dff42f6b6edfa3b92f29d05 in term 0.
I20260926 04:07:37.922878 1570 raft_consensus.cc:2454] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79effd792dff42f6b6edfa3b92f29d05 in term 0.
I20260926 04:07:37.923123 1773 leader_election.cc:302] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d01b683935342d3add6a5186ce6e250, 79effd792dff42f6b6edfa3b92f29d05; no voters:
I20260926 04:07:37.923333 1909 raft_consensus.cc:2781] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:37.923400 1909 raft_consensus.cc:482] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:37.923431 1909 raft_consensus.cc:3037] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:37.924273 1909 raft_consensus.cc:504] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.924430 1909 leader_election.cc:288] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [CANDIDATE]: Term 1 election: Requested vote from peers a07364af14c5458e95e31e44a1ffbb5b (127.0.213.193:45235), 2d01b683935342d3add6a5186ce6e250 (127.0.213.194:40159)
I20260926 04:07:37.924574 1570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602edbdf44524e0ab04eb60017f36814" candidate_uuid: "79effd792dff42f6b6edfa3b92f29d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07364af14c5458e95e31e44a1ffbb5b"
I20260926 04:07:37.924618 1704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "602edbdf44524e0ab04eb60017f36814" candidate_uuid: "79effd792dff42f6b6edfa3b92f29d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d01b683935342d3add6a5186ce6e250"
I20260926 04:07:37.924693 1570 raft_consensus.cc:3037] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:37.924723 1704 raft_consensus.cc:3037] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:37.925443 1570 raft_consensus.cc:2454] T 602edbdf44524e0ab04eb60017f36814 P a07364af14c5458e95e31e44a1ffbb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79effd792dff42f6b6edfa3b92f29d05 in term 1.
I20260926 04:07:37.925629 1773 leader_election.cc:302] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79effd792dff42f6b6edfa3b92f29d05, a07364af14c5458e95e31e44a1ffbb5b; no voters:
I20260926 04:07:37.925645 1704 raft_consensus.cc:2454] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79effd792dff42f6b6edfa3b92f29d05 in term 1.
I20260926 04:07:37.925766 1909 raft_consensus.cc:2781] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:37.925968 1909 raft_consensus.cc:686] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [term 1 LEADER]: Becoming Leader. State: Replica: 79effd792dff42f6b6edfa3b92f29d05, State: Running, Role: LEADER
I20260926 04:07:37.926088 1909 consensus_queue.cc:237] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } }
I20260926 04:07:37.927052 1424 catalog_manager.cc:5495] T 602edbdf44524e0ab04eb60017f36814 P 79effd792dff42f6b6edfa3b92f29d05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79effd792dff42f6b6edfa3b92f29d05 (127.0.213.195). New cstate: current_term: 1 leader_uuid: "79effd792dff42f6b6edfa3b92f29d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79effd792dff42f6b6edfa3b92f29d05" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 45681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a07364af14c5458e95e31e44a1ffbb5b" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 45235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d01b683935342d3add6a5186ce6e250" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40159 } health_report { overall_health: UNKNOWN } } }
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:37.939798 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.932039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.0.213.193:45235)
0926 04:07:37.932116 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.932118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.932244 (+ 126us) server_negotiation.cc:407] Connection header received
0926 04:07:37.932335 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.932335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.932344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.932393 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.932943 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.933444 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.934156 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.934370 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.934428 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.937021 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.937032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.937033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.937040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.938836 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.939161 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.939164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.939164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.939216 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.939374 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.939374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.939375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.939501 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.939599 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.939670 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.939729 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 04:07:37.940619 1751 tablet.cc:2369] T 602edbdf44524e0ab04eb60017f36814 P 2d01b683935342d3add6a5186ce6e250: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:37.964025 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.956312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.0.213.193:45235)
0926 04:07:37.956412 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.956414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.956489 (+ 75us) server_negotiation.cc:407] Connection header received
0926 04:07:37.956540 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.956541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.956550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.956602 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.957180 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.957664 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.958439 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.958635 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.958706 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.961236 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.961238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.961238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.961245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.962761 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.963124 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.963125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.963125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.963184 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.963441 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.963441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.963441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.963543 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.963635 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.963882 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.963954 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 04:07:37 dist-test-slave-xf24 krb5kdc[1380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:37.987306 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:37.979715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.0.213.193:45235)
0926 04:07:37.979784 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:37.979805 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:37.979839 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:07:37.979952 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:37.979953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:37.979968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:37.980031 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:37.980741 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.981224 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.981944 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:37.982123 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:37.982182 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:37.984681 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:37.984683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:37.984684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:37.984693 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:37.986137 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.986528 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.986529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.986530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.986585 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:37.986785 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:37.986786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:37.986786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:37.986890 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:37.987001 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:37.987103 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 04:07:37.987192 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 04:07:37.987807 1530 tablet_service.cc:2908] Rejecting scan request for tablet 602edbdf44524e0ab04eb60017f36814: Uninitialized: clean time has not yet been initialized
I20260926 04:07:37.989267 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1487
I20260926 04:07:37.996431 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1621
I20260926 04:07:38.002082 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1755
I20260926 04:07:38.006836 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1393
2026-09-26T04:07:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1969 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:38 dist-test-slave-xf24 krb5kdc[1925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:38 dist-test-slave-xf24 krb5kdc[1925](info): set up 2 sockets
Sep 26 04:07:38 dist-test-slave-xf24 krb5kdc[1925](info): commencing operation
krb5kdc: starting...
W20260926 04:07:39.067829 855 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:39.196259 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46795
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:39.277089 1938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:39.277347 1938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:39.277384 1938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:39.278591 1938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:39.278663 1938 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 04:07:39.278685 1938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:39.278707 1938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:39.278739 1938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:39.278762 1938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:39.280768 1938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46795
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:39.281038 1938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:39.281286 1938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:39.286253 1946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:39.286374 1938 server_base.cc:1034] running on GCE node
W20260926 04:07:39.286469 1943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:39.286296 1944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:39.286791 1938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:39.287052 1938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:39.288180 1938 hybrid_clock.cc:648] HybridClock initialized: now 1790395659288152 us; error 42 us; skew 500 ppm
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:39.294605 1938 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:39.295169 1938 webserver.cc:460] Webserver started at http://127.0.213.254:35177/ using document root <none> and password file <none>
I20260926 04:07:39.295429 1938 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:39.295499 1938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:39.295658 1938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:39.296993 1938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "fc65a1785196411ab044d3b45d86d7f9"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "c60a8ac63e4c1dc9e033dd5882a7a09c"
server_key_iv: "8b7b1ab02218b93daf0b55053eb3a692"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.297399 1938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "fc65a1785196411ab044d3b45d86d7f9"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "c60a8ac63e4c1dc9e033dd5882a7a09c"
server_key_iv: "8b7b1ab02218b93daf0b55053eb3a692"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.299158 1938 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 04:07:39.300218 1953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.300467 1938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:39.300611 1938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "fc65a1785196411ab044d3b45d86d7f9"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "c60a8ac63e4c1dc9e033dd5882a7a09c"
server_key_iv: "8b7b1ab02218b93daf0b55053eb3a692"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.300722 1938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:39.314342 1938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:39.316771 1938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:39.316962 1938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:39.322389 1938 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:46795
I20260926 04:07:39.322577 2015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:46795 every 8 connection(s)
I20260926 04:07:39.322882 1938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:39.323614 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:39.325985 2016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: Bootstrap starting.
I20260926 04:07:39.326608 2016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:39.326915 2016 log.cc:824] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:39.327628 2016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: No bootstrap required, opened a new log
I20260926 04:07:39.329874 2016 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc65a1785196411ab044d3b45d86d7f9" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46795 } }
I20260926 04:07:39.330027 2016 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:39.330050 2016 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc65a1785196411ab044d3b45d86d7f9, State: Initialized, Role: FOLLOWER
I20260926 04:07:39.330199 2016 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc65a1785196411ab044d3b45d86d7f9" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46795 } }
I20260926 04:07:39.330279 2016 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:39.330323 2016 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:39.330374 2016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:39.330912 2016 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc65a1785196411ab044d3b45d86d7f9" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46795 } }
I20260926 04:07:39.331027 2016 leader_election.cc:302] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc65a1785196411ab044d3b45d86d7f9; no voters:
I20260926 04:07:39.331215 2016 leader_election.cc:288] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:39.331265 2019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:39.331472 2019 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [term 1 LEADER]: Becoming Leader. State: Replica: fc65a1785196411ab044d3b45d86d7f9, State: Running, Role: LEADER
I20260926 04:07:39.331712 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 1938
I20260926 04:07:39.331816 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:39.331984 855 external_mini_cluster.cc:1428] Setting key ec20a0ec146637e3ca19f772a88d8ab6
I20260926 04:07:39.331485 2016 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:39.331584 2019 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc65a1785196411ab044d3b45d86d7f9" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46795 } }
I20260926 04:07:39.332808 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc65a1785196411ab044d3b45d86d7f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc65a1785196411ab044d3b45d86d7f9" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46795 } } }
I20260926 04:07:39.332892 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:39.333104 2021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc65a1785196411ab044d3b45d86d7f9. Latest consensus state: current_term: 1 leader_uuid: "fc65a1785196411ab044d3b45d86d7f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc65a1785196411ab044d3b45d86d7f9" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46795 } } }
I20260926 04:07:39.333170 2021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:39.334110 2025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:39.334637 2025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:39.336485 2025 catalog_manager.cc:1347] Generated new cluster ID: 3188b9e9e558429e89212e2912865f1f
I20260926 04:07:39.336542 2025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:39.343489 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.332546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.0.213.254:46795)
0926 04:07:39.332715 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.332727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.332768 (+ 41us) server_negotiation.cc:407] Connection header received
0926 04:07:39.332893 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.332900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.332977 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.333152 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.334151 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.335254 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.336170 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.336685 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.336805 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.339487 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.339491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.339492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.339505 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.341810 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.342232 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.342235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.342236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.342346 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.342580 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.342582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.342582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.342837 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.343018 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.343117 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.343263 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260926 04:07:39.349076 2025 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:39.349637 2025 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:39.353353 2025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: Generated new TSK 0
I20260926 04:07:39.353549 2025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:39.384407 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46795
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:39.490186 2042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:39.490409 2042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:39.490433 2042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:39.491672 2042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:39.491721 2042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:39.491777 2042 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:07:39.491808 2042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:39.493463 2042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:39.493745 2042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:39.493983 2042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:39.499248 2048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:39.499315 2050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:39.499325 2047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:39.499447 2042 server_base.cc:1034] running on GCE node
I20260926 04:07:39.499662 2042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:39.499925 2042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:39.501113 2042 hybrid_clock.cc:648] HybridClock initialized: now 1790395659501103 us; error 41 us; skew 500 ppm
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:39.503767 2042 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:39.504372 2042 webserver.cc:460] Webserver started at http://127.0.213.193:44687/ using document root <none> and password file <none>
I20260926 04:07:39.504616 2042 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:39.504662 2042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:39.504801 2042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:39.505914 2042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "5accc736c00a4184ab60b1aeb8d6676d"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "4ca37b966bad0972901b8844115f67a2"
server_key_iv: "8450bf65abe8558404e931f25265d904"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.506294 2042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "5accc736c00a4184ab60b1aeb8d6676d"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "4ca37b966bad0972901b8844115f67a2"
server_key_iv: "8450bf65abe8558404e931f25265d904"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.507735 2042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:39.508706 2057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.508908 2042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:39.509023 2042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "5accc736c00a4184ab60b1aeb8d6676d"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "4ca37b966bad0972901b8844115f67a2"
server_key_iv: "8450bf65abe8558404e931f25265d904"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.509102 2042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:39.526435 2042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:39.529075 2042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:39.529233 2042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:39.529804 2042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:39.529851 2042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.529902 2042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:39.529932 2042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.537134 2042 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:42057
I20260926 04:07:39.537228 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:42057 every 8 connection(s)
I20260926 04:07:39.537571 2042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:39.539498 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2042
I20260926 04:07:39.539572 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:39.539767 855 external_mini_cluster.cc:1428] Setting key 668951bc41872358ba31a26e3b754d88
I20260926 04:07:39.541388 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.530417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:42095 (local address 127.0.213.254:46795)
0926 04:07:39.530479 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.530481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.530689 (+ 208us) server_negotiation.cc:407] Connection header received
0926 04:07:39.531322 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.531323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.531341 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.531425 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.532454 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.533326 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.535077 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.535299 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.535357 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.538329 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.538330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.538331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.538340 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.539748 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.540256 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.540257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.540257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.540323 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.540612 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.540613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.540613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.540779 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.540863 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.541160 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.541253 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:07:39.541399 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.530393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.213.193:42095)
0926 04:07:39.530560 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.530596 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.531206 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.531555 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.531561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.531976 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:39.532319 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.532329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.533409 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.533410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.534982 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.534986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.535408 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.535409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.535568 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.536347 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:39.536357 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:39.538191 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:39.539841 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.539849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.539852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.540182 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.540391 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.540392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.540392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.540525 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.540877 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:39.540883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:39.541007 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.541099 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.541225 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260926 04:07:39.542047 2173 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46795
I20260926 04:07:39.542157 2173 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:39.542354 2173 heartbeater.cc:511] Master 127.0.213.254:46795 requested a full tablet report, sending...
I20260926 04:07:39.542865 1969 ts_manager.cc:194] Registered new tserver with Master: 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193:42057)
I20260926 04:07:39.543627 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:42095
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:07:39.550225 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.542999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.213.193:43973)
0926 04:07:39.543144 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.543156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.543209 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.544112 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.544114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.544332 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:39.544407 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.544410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.545113 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.545115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.546083 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.546087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.546472 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.546473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.546546 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.547048 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:39.547054 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:39.547463 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:39.549236 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.549237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.549237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.549403 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.549593 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.549594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.549594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.549647 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.549972 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:39.549972 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:39.550013 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.550072 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.550152 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260926 04:07:39.550227 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.543924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:43973 (local address 127.0.213.254:46795)
0926 04:07:39.543963 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.543965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.543972 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:07:39.543991 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.543992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.544002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.544047 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.544496 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.544998 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.546188 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.546383 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.546450 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.547564 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.547565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.547565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.547572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.549146 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.549475 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.549476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.549476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.549530 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.549701 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.549701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.549701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.549899 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.549969 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.550090 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.550164 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:39.567929 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46795
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:39.652670 2182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:39.652895 2182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:39.652920 2182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:39.654187 2182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:39.654234 2182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:39.654255 2182 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:07:39.654274 2182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:39.655815 2182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:39.656105 2182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:39.656359 2182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:39.661553 2187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:39.661553 2190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:39.661569 2188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:39.661605 2182 server_base.cc:1034] running on GCE node
I20260926 04:07:39.661962 2182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:39.662235 2182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:39.663440 2182 hybrid_clock.cc:648] HybridClock initialized: now 1790395659663413 us; error 46 us; skew 500 ppm
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:39.666213 2182 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:39.666730 2182 webserver.cc:460] Webserver started at http://127.0.213.194:41831/ using document root <none> and password file <none>
I20260926 04:07:39.667009 2182 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:39.667060 2182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:39.667197 2182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:39.668427 2182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "f2858713773c4b51b12205821bc87914"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "ed5f66cc1c8ed3555dd4a07577bfdc6b"
server_key_iv: "faccddaa7a4f1d82817da7fea2453b05"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.668807 2182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "f2858713773c4b51b12205821bc87914"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "ed5f66cc1c8ed3555dd4a07577bfdc6b"
server_key_iv: "faccddaa7a4f1d82817da7fea2453b05"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.670365 2182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:39.671237 2197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.671463 2182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:39.671574 2182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "f2858713773c4b51b12205821bc87914"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "ed5f66cc1c8ed3555dd4a07577bfdc6b"
server_key_iv: "faccddaa7a4f1d82817da7fea2453b05"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.671651 2182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:39.678968 2182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:39.681053 2182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:39.681219 2182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:39.681815 2182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:39.681866 2182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.681905 2182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:39.681919 2182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:39.689250 2182 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38689
I20260926 04:07:39.689335 2312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38689 every 8 connection(s)
I20260926 04:07:39.689654 2182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:39.691783 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.682280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:49891 (local address 127.0.213.254:46795)
0926 04:07:39.682351 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.682353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.682642 (+ 289us) server_negotiation.cc:407] Connection header received
0926 04:07:39.683438 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.683440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.683453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.683535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.684283 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.685125 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.686187 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.686389 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.686473 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.689049 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.689051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.689051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.689061 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.690642 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.691035 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.691036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.691036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.691086 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.691288 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.691289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.691289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.691391 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.691441 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.691662 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.691721 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:07:39.691987 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.682270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.213.194:49891)
0926 04:07:39.682457 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.682504 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.683275 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.683651 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.683655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.683937 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:39.684204 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.684209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.685258 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.685259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.686087 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.686089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.686490 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.686491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.686620 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.687220 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:39.687228 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:39.688946 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:39.690721 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.690726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.690728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.690964 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.691142 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.691142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.691142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.691233 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.691446 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:39.691450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:39.691558 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.691642 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.691760 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260926 04:07:39.692514 2313 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46795
I20260926 04:07:39.692610 2313 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:39.692842 2313 heartbeater.cc:511] Master 127.0.213.254:46795 requested a full tablet report, sending...
I20260926 04:07:39.692920 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2182
I20260926 04:07:39.693007 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:39.693187 855 external_mini_cluster.cc:1428] Setting key c7754ce636a4f97f77fe8a5f5d95f641
I20260926 04:07:39.693217 1970 ts_manager.cc:194] Registered new tserver with Master: f2858713773c4b51b12205821bc87914 (127.0.213.194:38689)
I20260926 04:07:39.693722 1970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:49891
I20260926 04:07:39.699983 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.693541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.213.194:54281)
0926 04:07:39.693666 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.693678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.693731 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.694010 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.694011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.694224 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:39.694297 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.694298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.694981 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.694982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.695589 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.695591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.696119 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.696120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.696178 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.696638 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:39.696648 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:39.696921 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:39.698690 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.698690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.698691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.698871 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.699072 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.699073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.699073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.699127 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.699470 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:39.699471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:39.699550 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.699669 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.699847 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260926 04:07:39.699992 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.693825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:54281 (local address 127.0.213.254:46795)
0926 04:07:39.693865 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.693867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.693877 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:39.693896 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.693897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.693907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.693967 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.694384 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.694881 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.695713 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.696012 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.696120 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.697008 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.697010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.697010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.697018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.698603 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.698942 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.698944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.698944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.699004 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.699188 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.699189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.699189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.699379 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.699455 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.699738 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.699845 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:39.719368 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46795
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:39.800400 2322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:39.800606 2322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:39.800630 2322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:39.801736 2322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:39.801774 2322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:39.801793 2322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:07:39.801810 2322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:39.803232 2322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:39.803447 2322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:39.803689 2322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:39.808873 2328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:39.808873 2327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:39.808941 2322 server_base.cc:1034] running on GCE node
W20260926 04:07:39.808874 2330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:39.809335 2322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:39.809617 2322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:39.810793 2322 hybrid_clock.cc:648] HybridClock initialized: now 1790395659810768 us; error 42 us; skew 500 ppm
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:39.813685 2322 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:39.814239 2322 webserver.cc:460] Webserver started at http://127.0.213.195:43483/ using document root <none> and password file <none>
I20260926 04:07:39.814472 2322 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:39.814540 2322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:39.814713 2322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:39.815805 2322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "9f7cda66c96b4bb892e70871c47912f8"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "36f74303139ed6de6815ac159c159380"
server_key_iv: "8597049109b4c8d2a9f2419249791cb4"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.816246 2322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "9f7cda66c96b4bb892e70871c47912f8"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "36f74303139ed6de6815ac159c159380"
server_key_iv: "8597049109b4c8d2a9f2419249791cb4"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.817652 2322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:39.818624 2337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.818921 2322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:39.819036 2322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "9f7cda66c96b4bb892e70871c47912f8"
format_stamp: "Formatted at 2026-09-26 04:07:39 on dist-test-slave-xf24"
server_key: "36f74303139ed6de6815ac159c159380"
server_key_iv: "8597049109b4c8d2a9f2419249791cb4"
server_key_version: "encryptionkey@0"
I20260926 04:07:39.819132 2322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:39.847755 2322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:39.850149 2322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:39.850353 2322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:39.850919 2322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:39.850986 2322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:39.851033 2322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:39.851056 2322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:39.859091 2322 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:44079
I20260926 04:07:39.859186 2452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:44079 every 8 connection(s)
I20260926 04:07:39.859541 2322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:39.861804 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.851559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:47259 (local address 127.0.213.254:46795)
0926 04:07:39.851615 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.851618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.851815 (+ 197us) server_negotiation.cc:407] Connection header received
0926 04:07:39.852425 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.852426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.852440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.852497 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.853538 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.854407 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.855452 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.855793 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.855887 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.858390 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.858391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.858392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.858401 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.860266 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.860723 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.860725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.860725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.860803 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.861074 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.861075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.861075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.861213 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.861289 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.861602 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.861710 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 04:07:39.861990 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.851461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.213.195:47259)
0926 04:07:39.851631 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.851722 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.852317 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.852578 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.852582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.853076 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:39.853404 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.853411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.854513 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.854515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.855375 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.855378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.855904 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.855905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.856109 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.856729 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:39.856738 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:39.858256 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:39.860374 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.860381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.860383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.860624 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.860878 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.860879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.860880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.861002 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.861291 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:39.861298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:39.861458 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.861566 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.861744 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260926 04:07:39.862593 2453 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46795
I20260926 04:07:39.862687 2453 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:39.862891 2453 heartbeater.cc:511] Master 127.0.213.254:46795 requested a full tablet report, sending...
I20260926 04:07:39.863318 1969 ts_manager.cc:194] Registered new tserver with Master: 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:39.863864 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:47259
I20260926 04:07:39.864424 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2322
I20260926 04:07:39.864507 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:39.864701 855 external_mini_cluster.cc:1428] Setting key 1cdd692939b4fcf4423f863fb63fb9aa
I20260926 04:07:39.865447 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:07:39.870155 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.863772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:60453 (local address 127.0.213.254:46795)
0926 04:07:39.863820 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.863822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.863887 (+ 65us) server_negotiation.cc:407] Connection header received
0926 04:07:39.863971 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.863971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.863983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.864042 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.864489 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.864995 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.865826 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.866027 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.866107 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.867252 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.867254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.867254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.867262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.868786 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.869193 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.869194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.869194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.869258 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.869475 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.869476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.869476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.869606 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.869691 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.869937 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.870047 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:07:39.870170 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.863672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.213.195:60453)
0926 04:07:39.863821 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.863833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.863898 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.864101 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.864102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.864323 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:39.864399 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.864401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.865095 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.865096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.865738 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.865741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.866105 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.866106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.866173 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.866714 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:39.866719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:39.867086 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:39.868896 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.868898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.868898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.869095 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.869337 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:39.869338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:39.869338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:39.869407 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:39.869680 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:39.869681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:39.869736 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.869829 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.870051 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 04:07:39 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:39.900813 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.892790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.0.213.254:46795)
0926 04:07:39.892858 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.892860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.892874 (+ 14us) server_negotiation.cc:407] Connection header received
0926 04:07:39.892910 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.892912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.892927 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.893021 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:39.893735 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.894594 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.895362 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.895644 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.895730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.898184 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:39.898185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:39.898186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:39.898193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:39.899502 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.899926 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.899928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.899928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.900000 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:39.900255 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:39.900256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:39.900256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:39.900376 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:39.900460 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.900610 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.900679 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 04:07:39.902532 1969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:39.904296 1969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:39.911228 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.906899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.0.213.194:38689)
0926 04:07:39.907071 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.907076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.907088 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:07:39.907130 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.907132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.907149 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.907255 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:39.907529 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.908742 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910069 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.910749 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910829 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.910908 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.911058 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.911100 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260926 04:07:39.911201 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.907211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:42057 (local address 127.0.0.1:48692)
0926 04:07:39.907567 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.907576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.907631 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.907837 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.907838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.907840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:39.907886 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.907888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.908811 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.908812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.910127 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.910130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.910991 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.911052 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.911054 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.911099 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.911133 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":38,"thread_start_us":62,"threads_started":1}
I20260926 04:07:39.911523 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.907027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:44079 (local address 127.0.0.1:48068)
0926 04:07:39.907158 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.907169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.907220 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.907915 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.907916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.907919 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:39.907967 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.907968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.909181 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.909182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.910264 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.910266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.911287 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.911288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.911362 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.911366 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.911418 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.911460 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260926 04:07:39.911201 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.906787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38689 (local address 127.0.0.1:40238)
0926 04:07:39.906911 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:39.906956 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:39.907021 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:39.907339 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:39.907341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:39.907343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:39.907434 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.907436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.908837 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.908838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.909981 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:39.909983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.910847 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:39.910938 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.910955 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 04:07:39.911032 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 04:07:39.911097 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":51,"thread_start_us":78,"threads_started":1}
I20260926 04:07:39.911793 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.907433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.0.213.195:44079)
0926 04:07:39.907656 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.907660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.907671 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:07:39.907760 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.907763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.907780 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.907850 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:39.908030 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.909081 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910343 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.911105 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.911178 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.911248 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.911447 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.911486 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260926 04:07:39.912415 2247 tablet_service.cc:1467] Processing CreateTablet for tablet 320a122f37a349dfbd5db8dd77900198 (DEFAULT_TABLE table=test-table [id=ed29a89617b348d9ba798f0feb5e21d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:39.912415 2380 tablet_service.cc:1467] Processing CreateTablet for tablet 320a122f37a349dfbd5db8dd77900198 (DEFAULT_TABLE table=test-table [id=ed29a89617b348d9ba798f0feb5e21d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:39.912791 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320a122f37a349dfbd5db8dd77900198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:39.912793 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320a122f37a349dfbd5db8dd77900198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:39.915769 2476 tablet_bootstrap.cc:492] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Bootstrap starting.
I20260926 04:07:39.916051 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:39.907345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.0.213.193:42057)
0926 04:07:39.907520 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:39.907525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:39.907690 (+ 165us) server_negotiation.cc:407] Connection header received
0926 04:07:39.907728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:39.907730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:39.907744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:39.907825 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:39.907956 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.908707 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910238 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:39.910788 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:39.910881 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:39.910959 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:39.911741 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 04:07:39.911785 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260926 04:07:39.916113 2477 tablet_bootstrap.cc:492] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Bootstrap starting.
I20260926 04:07:39.916512 2476 tablet_bootstrap.cc:654] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:39.916855 2476 log.cc:824] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:39.916838 2107 tablet_service.cc:1467] Processing CreateTablet for tablet 320a122f37a349dfbd5db8dd77900198 (DEFAULT_TABLE table=test-table [id=ed29a89617b348d9ba798f0feb5e21d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:39.916985 2477 tablet_bootstrap.cc:654] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:39.917196 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320a122f37a349dfbd5db8dd77900198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:39.917287 2477 log.cc:824] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:39.918090 2477 tablet_bootstrap.cc:492] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: No bootstrap required, opened a new log
I20260926 04:07:39.918166 2477 ts_tablet_manager.cc:1397] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:39.920455 2476 tablet_bootstrap.cc:492] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: No bootstrap required, opened a new log
I20260926 04:07:39.920406 2477 raft_consensus.cc:348] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:39.920553 2476 ts_tablet_manager.cc:1397] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 04:07:39.920588 2477 raft_consensus.cc:374] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:39.920634 2477 raft_consensus.cc:729] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2858713773c4b51b12205821bc87914, State: Initialized, Role: FOLLOWER
I20260926 04:07:39.920770 2477 consensus_queue.cc:260] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:39.921048 2477 ts_tablet_manager.cc:1428] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:39.921182 2313 heartbeater.cc:503] Master 127.0.213.254:46795 was elected leader, sending a full tablet report...
I20260926 04:07:39.921085 2480 tablet_bootstrap.cc:492] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Bootstrap starting.
I20260926 04:07:39.922664 2480 tablet_bootstrap.cc:654] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:39.922873 2476 raft_consensus.cc:348] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:39.923053 2476 raft_consensus.cc:374] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:39.923053 2480 log.cc:824] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:39.923101 2476 raft_consensus.cc:729] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7cda66c96b4bb892e70871c47912f8, State: Initialized, Role: FOLLOWER
I20260926 04:07:39.923229 2476 consensus_queue.cc:260] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:39.923534 2476 ts_tablet_manager.cc:1428] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 04:07:39.923565 2453 heartbeater.cc:503] Master 127.0.213.254:46795 was elected leader, sending a full tablet report...
I20260926 04:07:39.924274 2480 tablet_bootstrap.cc:492] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: No bootstrap required, opened a new log
I20260926 04:07:39.924346 2480 ts_tablet_manager.cc:1397] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:07:39.925978 2480 raft_consensus.cc:348] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:39.926141 2480 raft_consensus.cc:374] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:39.926188 2480 raft_consensus.cc:729] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5accc736c00a4184ab60b1aeb8d6676d, State: Initialized, Role: FOLLOWER
I20260926 04:07:39.926306 2480 consensus_queue.cc:260] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:39.926592 2480 ts_tablet_manager.cc:1428] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:39.926626 2173 heartbeater.cc:503] Master 127.0.213.254:46795 was elected leader, sending a full tablet report...
W20260926 04:07:39.940382 2314 tablet.cc:2369] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:07:40.038347 2174 tablet.cc:2369] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:07:40.071942 2484 raft_consensus.cc:482] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:40.072094 2484 raft_consensus.cc:504] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:40.072512 2484 leader_election.cc:288] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2858713773c4b51b12205821bc87914 (127.0.213.194:38689), 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:40.075958 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.072848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:37819 (local address 127.0.213.194:38689)
0926 04:07:40.072916 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.072918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.072982 (+ 64us) server_negotiation.cc:407] Connection header received
0926 04:07:40.073122 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.073123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.073135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.073227 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:40.073490 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.074075 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.075138 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.075570 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.075625 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.075632 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.075837 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.075873 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:07:40.075999 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.072759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38689 (local address 127.0.213.193:37819)
0926 04:07:40.072893 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.072906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.072984 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.073239 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.073240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.073243 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:40.073316 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.073319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.074185 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.074187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.075070 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.075072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.075687 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.075688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.075756 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.075761 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.075819 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.075870 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":40,"thread_start_us":67,"threads_started":1}
I20260926 04:07:40.076145 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.073065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:38633 (local address 127.0.213.195:44079)
0926 04:07:40.073129 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.073131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.073159 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:07:40.073237 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.073238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.073247 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.073305 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:40.073593 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.074144 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.075268 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.075752 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.075828 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.075835 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.075990 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.076035 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:07:40.076261 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.072947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:44079 (local address 127.0.213.193:38633)
0926 04:07:40.073059 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.073070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.073154 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.073330 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.073331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.073333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:40.073515 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.073517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.074221 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.074222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.075145 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.075148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.075842 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.075843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.075909 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.075912 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.075970 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.076014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260926 04:07:40.076661 2267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320a122f37a349dfbd5db8dd77900198" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2858713773c4b51b12205821bc87914" is_pre_election: true
I20260926 04:07:40.076649 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320a122f37a349dfbd5db8dd77900198" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7cda66c96b4bb892e70871c47912f8" is_pre_election: true
I20260926 04:07:40.076829 2406 raft_consensus.cc:2454] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 0.
I20260926 04:07:40.076829 2267 raft_consensus.cc:2454] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 0.
I20260926 04:07:40.077054 2061 leader_election.cc:302] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5accc736c00a4184ab60b1aeb8d6676d, f2858713773c4b51b12205821bc87914; no voters:
I20260926 04:07:40.077184 2484 raft_consensus.cc:2781] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:40.077239 2484 raft_consensus.cc:482] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:40.077267 2484 raft_consensus.cc:3037] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.077940 2484 raft_consensus.cc:504] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:40.078078 2484 leader_election.cc:288] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 election: Requested vote from peers f2858713773c4b51b12205821bc87914 (127.0.213.194:38689), 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:40.078238 2267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320a122f37a349dfbd5db8dd77900198" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2858713773c4b51b12205821bc87914"
I20260926 04:07:40.078271 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320a122f37a349dfbd5db8dd77900198" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7cda66c96b4bb892e70871c47912f8"
I20260926 04:07:40.078310 2267 raft_consensus.cc:3037] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.078369 2406 raft_consensus.cc:3037] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.078989 2267 raft_consensus.cc:2454] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 1.
I20260926 04:07:40.079068 2406 raft_consensus.cc:2454] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 1.
I20260926 04:07:40.079210 2061 leader_election.cc:302] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5accc736c00a4184ab60b1aeb8d6676d, f2858713773c4b51b12205821bc87914; no voters:
I20260926 04:07:40.079312 2484 raft_consensus.cc:2781] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:40.079411 2484 raft_consensus.cc:686] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 LEADER]: Becoming Leader. State: Replica: 5accc736c00a4184ab60b1aeb8d6676d, State: Running, Role: LEADER
I20260926 04:07:40.079532 2484 consensus_queue.cc:237] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } }
I20260926 04:07:40.080436 1969 catalog_manager.cc:5495] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193). New cstate: current_term: 1 leader_uuid: "5accc736c00a4184ab60b1aeb8d6676d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } health_report { overall_health: HEALTHY } } }
I20260926 04:07:40.093963 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.091073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.0.213.193:42057)
0926 04:07:40.091150 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.091152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.091162 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:40.091259 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.091264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.091282 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.091368 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:40.091540 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.092038 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.093002 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.093260 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.093337 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.093577 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.093701 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.093798 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.093850 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.093872 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:07:40.095809 2267 raft_consensus.cc:1261] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Refusing update from remote peer 5accc736c00a4184ab60b1aeb8d6676d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:40.095817 2406 raft_consensus.cc:1261] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Refusing update from remote peer 5accc736c00a4184ab60b1aeb8d6676d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:40.096230 2484 consensus_queue.cc:1035] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:40.096372 2487 consensus_queue.cc:1035] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:40.102736 1969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:40.102867 1969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:40.103772 1969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: Sending DeleteTablet for 3 replicas of tablet 320a122f37a349dfbd5db8dd77900198
I20260926 04:07:40.104182 2380 tablet_service.cc:1514] Processing DeleteTablet for tablet 320a122f37a349dfbd5db8dd77900198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:48068
I20260926 04:07:40.104228 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet 320a122f37a349dfbd5db8dd77900198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40238
I20260926 04:07:40.104429 2500 tablet_replica.cc:330] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: stopping tablet replica
I20260926 04:07:40.104420 1969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 04:07:40.104538 2500 raft_consensus.cc:2229] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 04:07:40.104592 1969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:40.104590 2107 tablet_service.cc:1514] Processing DeleteTablet for tablet 320a122f37a349dfbd5db8dd77900198 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:48692
I20260926 04:07:40.104773 2500 raft_consensus.cc:2258] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:40.105433 2500 ts_tablet_manager.cc:1905] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:40.105701 2502 tablet_replica.cc:330] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: stopping tablet replica
I20260926 04:07:40.105863 2502 raft_consensus.cc:2229] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:40.106719 2247 tablet_service.cc:1467] Processing CreateTablet for tablet b562ef7a08384a18a07c2890851e8058 (DEFAULT_TABLE table=test-table [id=9f43694f36cd459d94467b6e8133e567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:40.106863 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b562ef7a08384a18a07c2890851e8058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:40.107798 2500 ts_tablet_manager.cc:1918] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:40.107899 2500 log.cc:1196] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/320a122f37a349dfbd5db8dd77900198
I20260926 04:07:40.108337 2500 ts_tablet_manager.cc:1939] T 320a122f37a349dfbd5db8dd77900198 P 9f7cda66c96b4bb892e70871c47912f8: Deleting consensus metadata
I20260926 04:07:40.108363 2380 tablet_service.cc:1467] Processing CreateTablet for tablet b562ef7a08384a18a07c2890851e8058 (DEFAULT_TABLE table=test-table [id=9f43694f36cd459d94467b6e8133e567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:40.108454 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b562ef7a08384a18a07c2890851e8058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:40.108588 2477 tablet_bootstrap.cc:492] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Bootstrap starting.
I20260926 04:07:40.108851 1957 catalog_manager.cc:4841] TS 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079): tablet 320a122f37a349dfbd5db8dd77900198 (table test-table [id=ed29a89617b348d9ba798f0feb5e21d4]) successfully deleted
I20260926 04:07:40.109022 2477 tablet_bootstrap.cc:654] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:40.109568 2476 tablet_bootstrap.cc:492] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Bootstrap starting.
I20260926 04:07:40.106559 2107 tablet_service.cc:1467] Processing CreateTablet for tablet b562ef7a08384a18a07c2890851e8058 (DEFAULT_TABLE table=test-table [id=9f43694f36cd459d94467b6e8133e567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:40.109824 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b562ef7a08384a18a07c2890851e8058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:40.109975 2476 tablet_bootstrap.cc:654] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:40.110590 2477 tablet_bootstrap.cc:492] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: No bootstrap required, opened a new log
I20260926 04:07:40.110600 2476 tablet_bootstrap.cc:492] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: No bootstrap required, opened a new log
I20260926 04:07:40.110643 2476 ts_tablet_manager.cc:1397] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:40.110644 2477 ts_tablet_manager.cc:1397] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:40.110785 2476 raft_consensus.cc:348] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.110852 2476 raft_consensus.cc:374] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:40.110818 2477 raft_consensus.cc:348] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.110873 2476 raft_consensus.cc:729] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7cda66c96b4bb892e70871c47912f8, State: Initialized, Role: FOLLOWER
I20260926 04:07:40.110893 2477 raft_consensus.cc:374] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:40.110914 2477 raft_consensus.cc:729] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2858713773c4b51b12205821bc87914, State: Initialized, Role: FOLLOWER
I20260926 04:07:40.110916 2476 consensus_queue.cc:260] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.110963 2477 consensus_queue.cc:260] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.110996 2476 ts_tablet_manager.cc:1428] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:40.111060 2477 ts_tablet_manager.cc:1428] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:40.111449 2480 tablet_bootstrap.cc:492] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Bootstrap starting.
I20260926 04:07:40.112308 2480 tablet_bootstrap.cc:654] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:40.112733 2501 tablet_replica.cc:330] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: stopping tablet replica
I20260926 04:07:40.111423 2502 raft_consensus.cc:2258] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:40.113130 2480 tablet_bootstrap.cc:492] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: No bootstrap required, opened a new log
I20260926 04:07:40.113184 2480 ts_tablet_manager.cc:1397] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 04:07:40.113334 2480 raft_consensus.cc:348] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.113444 2480 raft_consensus.cc:374] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:40.113471 2480 raft_consensus.cc:729] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5accc736c00a4184ab60b1aeb8d6676d, State: Initialized, Role: FOLLOWER
I20260926 04:07:40.113517 2480 consensus_queue.cc:260] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.113621 2480 ts_tablet_manager.cc:1428] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:40.113776 2501 raft_consensus.cc:2229] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:40.113909 2501 raft_consensus.cc:2258] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:40.114451 2502 ts_tablet_manager.cc:1905] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:40.115901 2502 ts_tablet_manager.cc:1918] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:40.116043 2502 log.cc:1196] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/320a122f37a349dfbd5db8dd77900198
I20260926 04:07:40.116081 2501 ts_tablet_manager.cc:1905] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:40.116417 2502 ts_tablet_manager.cc:1939] T 320a122f37a349dfbd5db8dd77900198 P 5accc736c00a4184ab60b1aeb8d6676d: Deleting consensus metadata
I20260926 04:07:40.116914 1957 catalog_manager.cc:4841] TS 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193:42057): tablet 320a122f37a349dfbd5db8dd77900198 (table test-table [id=ed29a89617b348d9ba798f0feb5e21d4]) successfully deleted
I20260926 04:07:40.117699 2501 ts_tablet_manager.cc:1918] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:40.117794 2501 log.cc:1196] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/320a122f37a349dfbd5db8dd77900198
I20260926 04:07:40.118093 2501 ts_tablet_manager.cc:1939] T 320a122f37a349dfbd5db8dd77900198 P f2858713773c4b51b12205821bc87914: Deleting consensus metadata
I20260926 04:07:40.118525 1957 catalog_manager.cc:4841] TS f2858713773c4b51b12205821bc87914 (127.0.213.194:38689): tablet 320a122f37a349dfbd5db8dd77900198 (table test-table [id=ed29a89617b348d9ba798f0feb5e21d4]) successfully deleted
I20260926 04:07:40.279500 2497 raft_consensus.cc:482] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:40.279652 2497 raft_consensus.cc:504] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.279924 2497 leader_election.cc:288] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2858713773c4b51b12205821bc87914 (127.0.213.194:38689), 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:40.280274 2267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b562ef7a08384a18a07c2890851e8058" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2858713773c4b51b12205821bc87914" is_pre_election: true
I20260926 04:07:40.280321 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b562ef7a08384a18a07c2890851e8058" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7cda66c96b4bb892e70871c47912f8" is_pre_election: true
I20260926 04:07:40.280390 2267 raft_consensus.cc:2454] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 0.
I20260926 04:07:40.280401 2406 raft_consensus.cc:2454] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 0.
I20260926 04:07:40.280567 2061 leader_election.cc:302] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5accc736c00a4184ab60b1aeb8d6676d, 9f7cda66c96b4bb892e70871c47912f8; no voters:
I20260926 04:07:40.280685 2497 raft_consensus.cc:2781] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:40.280737 2497 raft_consensus.cc:482] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:40.280769 2497 raft_consensus.cc:3037] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.281510 2497 raft_consensus.cc:504] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.281661 2497 leader_election.cc:288] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 election: Requested vote from peers f2858713773c4b51b12205821bc87914 (127.0.213.194:38689), 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:40.281858 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b562ef7a08384a18a07c2890851e8058" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7cda66c96b4bb892e70871c47912f8"
I20260926 04:07:40.281855 2267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b562ef7a08384a18a07c2890851e8058" candidate_uuid: "5accc736c00a4184ab60b1aeb8d6676d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2858713773c4b51b12205821bc87914"
I20260926 04:07:40.281955 2406 raft_consensus.cc:3037] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.281952 2267 raft_consensus.cc:3037] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.282646 2406 raft_consensus.cc:2454] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 1.
I20260926 04:07:40.282716 2267 raft_consensus.cc:2454] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5accc736c00a4184ab60b1aeb8d6676d in term 1.
I20260926 04:07:40.282855 2061 leader_election.cc:302] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5accc736c00a4184ab60b1aeb8d6676d, 9f7cda66c96b4bb892e70871c47912f8; no voters:
I20260926 04:07:40.282958 2497 raft_consensus.cc:2781] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:40.283018 2497 raft_consensus.cc:686] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 LEADER]: Becoming Leader. State: Replica: 5accc736c00a4184ab60b1aeb8d6676d, State: Running, Role: LEADER
I20260926 04:07:40.283100 2497 consensus_queue.cc:237] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.283701 1969 catalog_manager.cc:5495] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193). New cstate: current_term: 1 leader_uuid: "5accc736c00a4184ab60b1aeb8d6676d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } health_report { overall_health: UNKNOWN } } }
Sep 26 04:07:40 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:40.325584 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.316852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46795 (local address 127.0.0.1:46370)
0926 04:07:40.317035 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.317044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.317121 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.317329 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.317329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.317689 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:40.317755 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.317757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.318582 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.318583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.319144 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.319146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.319688 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.319689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.319743 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.320538 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:40.320543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:40.322169 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:40.324278 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:40.324284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:40.324286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:40.324583 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:40.324852 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:40.324853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:40.324854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:40.324926 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:40.325264 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:40.325271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:40.325321 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.325389 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.325493 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260926 04:07:40.325682 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.316951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.213.254:46795)
0926 04:07:40.317021 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.317023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.317174 (+ 151us) server_negotiation.cc:407] Connection header received
0926 04:07:40.317202 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.317202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.317225 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.317277 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:40.317885 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.318473 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.319281 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.319471 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.319624 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.322337 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:40.322338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:40.322339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:40.322348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:40.324180 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:40.324694 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:40.324696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:40.324696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:40.324769 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:40.325002 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:40.325003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:40.325003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:40.325172 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:40.325241 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.325436 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.325558 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":34}
I20260926 04:07:40.327808 1969 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:46370:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 04:07:40.328032 1969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:40.329479 2107 tablet_service.cc:1467] Processing CreateTablet for tablet ac3fb64ce52c411b8e33530c0ee8185a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=76e33733cfaf404baf4a8303545f4d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:07:40.329500 2247 tablet_service.cc:1467] Processing CreateTablet for tablet ac3fb64ce52c411b8e33530c0ee8185a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=76e33733cfaf404baf4a8303545f4d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:07:40.329536 2380 tablet_service.cc:1467] Processing CreateTablet for tablet ac3fb64ce52c411b8e33530c0ee8185a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=76e33733cfaf404baf4a8303545f4d47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:07:40.329682 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3fb64ce52c411b8e33530c0ee8185a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:40.329687 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3fb64ce52c411b8e33530c0ee8185a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:40.329701 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3fb64ce52c411b8e33530c0ee8185a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:40.330813 2477 tablet_bootstrap.cc:492] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: Bootstrap starting.
I20260926 04:07:40.331163 2476 tablet_bootstrap.cc:492] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: Bootstrap starting.
I20260926 04:07:40.331233 2477 tablet_bootstrap.cc:654] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:40.331578 2476 tablet_bootstrap.cc:654] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:40.331904 2477 tablet_bootstrap.cc:492] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: No bootstrap required, opened a new log
I20260926 04:07:40.331910 2480 tablet_bootstrap.cc:492] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: Bootstrap starting.
I20260926 04:07:40.331965 2477 ts_tablet_manager.cc:1397] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:40.332167 2477 raft_consensus.cc:348] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.332283 2477 raft_consensus.cc:374] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:40.332311 2477 raft_consensus.cc:729] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2858713773c4b51b12205821bc87914, State: Initialized, Role: FOLLOWER
I20260926 04:07:40.332363 2480 tablet_bootstrap.cc:654] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:40.332356 2477 consensus_queue.cc:260] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.332844 2476 tablet_bootstrap.cc:492] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: No bootstrap required, opened a new log
I20260926 04:07:40.332864 2477 ts_tablet_manager.cc:1428] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:40.332894 2476 ts_tablet_manager.cc:1397] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 04:07:40.332943 2480 tablet_bootstrap.cc:492] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: No bootstrap required, opened a new log
I20260926 04:07:40.333002 2480 ts_tablet_manager.cc:1397] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 04:07:40.333016 2481 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.333099 2481 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:40.333047 2476 raft_consensus.cc:348] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.333173 2476 raft_consensus.cc:374] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:40.333140 2480 raft_consensus.cc:348] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.333199 2476 raft_consensus.cc:729] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7cda66c96b4bb892e70871c47912f8, State: Initialized, Role: FOLLOWER
I20260926 04:07:40.333204 2480 raft_consensus.cc:374] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:40.333222 2480 raft_consensus.cc:729] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5accc736c00a4184ab60b1aeb8d6676d, State: Initialized, Role: FOLLOWER
I20260926 04:07:40.333240 2476 consensus_queue.cc:260] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.333261 2480 consensus_queue.cc:260] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.333338 2476 ts_tablet_manager.cc:1428] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:40.333369 2480 ts_tablet_manager.cc:1428] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:40.333536 2483 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.333655 2483 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:40.333765 2497 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.333858 2497 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 04:07:40.360342 2454 tablet.cc:2369] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:07:40.383994 2481 raft_consensus.cc:482] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:40.384131 2481 raft_consensus.cc:504] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.384493 2481 leader_election.cc:288] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193:42057), 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:40.389093 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.385020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:47957 (local address 127.0.213.195:44079)
0926 04:07:40.385065 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.385067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.385137 (+ 70us) server_negotiation.cc:407] Connection header received
0926 04:07:40.385215 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.385215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.385229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.385272 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:40.385440 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.386136 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.388000 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.388613 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.388721 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.388733 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.388972 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.389016 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:07:40.389093 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.384825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:43081 (local address 127.0.213.193:42057)
0926 04:07:40.384891 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.384893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.385001 (+ 108us) server_negotiation.cc:407] Connection header received
0926 04:07:40.385034 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.385035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.385048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.385130 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:40.385352 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.386125 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.387999 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.388635 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.388727 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.388734 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.388973 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.389014 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:07:40.389138 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.384726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:42057 (local address 127.0.213.194:43081)
0926 04:07:40.384874 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.384886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.384956 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.385147 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.385148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.385150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:40.385251 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.385271 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.386288 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.386290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.387834 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.387839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.388775 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.388776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.388873 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.388880 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.388947 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.389010 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":108,"thread_start_us":98,"threads_started":1}
I20260926 04:07:40.389139 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.384944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:44079 (local address 127.0.213.194:47957)
0926 04:07:40.385061 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.385075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.385146 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.385295 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.385296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.385298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:40.385356 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.385357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.386289 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.386290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.387834 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.387839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.388773 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.388775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.388873 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.388882 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.388947 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.389010 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":119,"thread_start_us":40,"threads_started":1}
I20260926 04:07:40.389497 2127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3fb64ce52c411b8e33530c0ee8185a" candidate_uuid: "f2858713773c4b51b12205821bc87914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5accc736c00a4184ab60b1aeb8d6676d" is_pre_election: true
I20260926 04:07:40.389529 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3fb64ce52c411b8e33530c0ee8185a" candidate_uuid: "f2858713773c4b51b12205821bc87914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7cda66c96b4bb892e70871c47912f8" is_pre_election: true
I20260926 04:07:40.389683 2406 raft_consensus.cc:2454] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2858713773c4b51b12205821bc87914 in term 0.
I20260926 04:07:40.389726 2127 raft_consensus.cc:2454] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2858713773c4b51b12205821bc87914 in term 0.
I20260926 04:07:40.389899 2201 leader_election.cc:302] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f7cda66c96b4bb892e70871c47912f8, f2858713773c4b51b12205821bc87914; no voters:
I20260926 04:07:40.390046 2481 raft_consensus.cc:2781] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:40.390094 2481 raft_consensus.cc:482] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:40.390118 2481 raft_consensus.cc:3037] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.390942 2481 raft_consensus.cc:504] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.391086 2481 leader_election.cc:288] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [CANDIDATE]: Term 1 election: Requested vote from peers 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193:42057), 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079)
I20260926 04:07:40.391240 2127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3fb64ce52c411b8e33530c0ee8185a" candidate_uuid: "f2858713773c4b51b12205821bc87914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5accc736c00a4184ab60b1aeb8d6676d"
I20260926 04:07:40.391258 2406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3fb64ce52c411b8e33530c0ee8185a" candidate_uuid: "f2858713773c4b51b12205821bc87914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7cda66c96b4bb892e70871c47912f8"
I20260926 04:07:40.391345 2127 raft_consensus.cc:3037] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.391343 2406 raft_consensus.cc:3037] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:40.392016 2406 raft_consensus.cc:2454] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2858713773c4b51b12205821bc87914 in term 1.
I20260926 04:07:40.392035 2127 raft_consensus.cc:2454] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2858713773c4b51b12205821bc87914 in term 1.
I20260926 04:07:40.392225 2201 leader_election.cc:302] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5accc736c00a4184ab60b1aeb8d6676d, f2858713773c4b51b12205821bc87914; no voters:
I20260926 04:07:40.392349 2481 raft_consensus.cc:2781] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:40.392496 2481 raft_consensus.cc:686] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [term 1 LEADER]: Becoming Leader. State: Replica: f2858713773c4b51b12205821bc87914, State: Running, Role: LEADER
I20260926 04:07:40.392589 2481 consensus_queue.cc:237] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } }
I20260926 04:07:40.392992 2518 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: TxnStatusTablet state changed. Reason: New leader f2858713773c4b51b12205821bc87914. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.393116 2518 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:07:40.393328 2520 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 04:07:40.393464 2520 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 04:07:40.393466 1970 catalog_manager.cc:5495] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2858713773c4b51b12205821bc87914 (127.0.213.194). New cstate: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:40.411010 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.408329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.0.213.194:38689)
0926 04:07:40.408393 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.408395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.408405 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:40.408580 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.408581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.408599 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.408670 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:40.408840 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.409340 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.410133 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.410341 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.410421 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.410631 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.410737 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.410821 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.410880 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.410902 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:07:40.410992 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.408255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38689 (local address 127.0.0.1:40244)
0926 04:07:40.408333 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.408365 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.408423 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.408688 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.408689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.408691 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:40.408763 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.408766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.409433 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.409434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.410045 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.410046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.410419 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.410420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.410503 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.410521 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.410810 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.410819 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.410866 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.410918 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:07:40.412449 2406 raft_consensus.cc:1261] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Refusing update from remote peer f2858713773c4b51b12205821bc87914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:40.412593 2127 raft_consensus.cc:1261] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Refusing update from remote peer f2858713773c4b51b12205821bc87914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:40.412779 2518 consensus_queue.cc:1035] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:40.412935 2481 consensus_queue.cc:1035] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:40.413548 2497 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: TxnStatusTablet state changed. Reason: New leader f2858713773c4b51b12205821bc87914. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.413650 2497 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:40.413954 2483 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: TxnStatusTablet state changed. Reason: New leader f2858713773c4b51b12205821bc87914. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.414052 2483 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:40.414497 2481 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.414600 2481 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:07:40.414572 2497 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.414641 2497 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P 5accc736c00a4184ab60b1aeb8d6676d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:40.414800 2490 mvcc.cc:205] Tried to move back new op lower bound from 7333460625047433216 to 7333460624968884224. Current Snapshot: MvccSnapshot[applied={T|T < 7333460625047433216 or (T in {7333460625047433216})}]
I20260926 04:07:40.415810 2518 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.415896 2518 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P f2858713773c4b51b12205821bc87914: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:07:40.416006 2492 mvcc.cc:205] Tried to move back new op lower bound from 7333460625047433216 to 7333460624968884224. Current Snapshot: MvccSnapshot[applied={T|T < 7333460625047433216}]
I20260926 04:07:40.416123 2483 tablet_replica.cc:439] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2858713773c4b51b12205821bc87914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5accc736c00a4184ab60b1aeb8d6676d" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 42057 } } peers { permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 } } peers { permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 } } }
I20260926 04:07:40.416196 2483 tablet_replica.cc:442] T ac3fb64ce52c411b8e33530c0ee8185a P 9f7cda66c96b4bb892e70871c47912f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:40.421451 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.418799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:58041 (local address 127.0.213.194:38689)
0926 04:07:40.418843 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.418845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.418922 (+ 77us) server_negotiation.cc:407] Connection header received
0926 04:07:40.418998 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.418998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.419006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.419046 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:40.419168 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.419629 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.420770 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.420949 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.421010 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.421213 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.421245 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.421299 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.421376 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.421394 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:07:40.421548 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.418715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38689 (local address 127.0.213.193:58041)
0926 04:07:40.418865 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.418877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.418939 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.419061 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.419062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.419064 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:40.419115 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.419116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.419704 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.419705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.420681 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.420683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.421028 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.421029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.421088 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.421104 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.421316 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.421318 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.421373 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.421459 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 04:07:40.423786 2407 raft_consensus.cc:1261] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Refusing update from remote peer 5accc736c00a4184ab60b1aeb8d6676d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:40.423838 2267 raft_consensus.cc:1261] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Refusing update from remote peer 5accc736c00a4184ab60b1aeb8d6676d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:40.424016 2497 consensus_queue.cc:1035] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2858713773c4b51b12205821bc87914" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:40.424150 2484 consensus_queue.cc:1035] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7cda66c96b4bb892e70871c47912f8" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:40.434443 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.430954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:41415 (local address 127.0.213.193:42057)
0926 04:07:40.431004 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.431006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.431123 (+ 117us) server_negotiation.cc:407] Connection header received
0926 04:07:40.431250 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.431251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.431261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.431306 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:40.431456 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.432018 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.433522 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.433842 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.433933 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.434136 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.434176 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.434261 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.434327 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.434351 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 04:07:40.434481 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.430860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:42057 (local address 127.0.213.194:41415)
0926 04:07:40.431047 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:40.431060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:40.431152 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:40.431320 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:40.431321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:40.431322 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:40.431393 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.431394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.432202 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.432203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.433409 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:40.433418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.433932 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.433933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:40.434004 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.434030 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:40.434260 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:40.434262 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:40.434322 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 04:07:40.434365 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260926 04:07:40.443743 1969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:40.443866 1969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:40.444581 1969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc65a1785196411ab044d3b45d86d7f9: Sending DeleteTablet for 3 replicas of tablet b562ef7a08384a18a07c2890851e8058
I20260926 04:07:40.444871 2247 tablet_service.cc:1514] Processing DeleteTablet for tablet b562ef7a08384a18a07c2890851e8058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40238
I20260926 04:07:40.444900 2107 tablet_service.cc:1514] Processing DeleteTablet for tablet b562ef7a08384a18a07c2890851e8058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:48692
I20260926 04:07:40.444923 2380 tablet_service.cc:1514] Processing DeleteTablet for tablet b562ef7a08384a18a07c2890851e8058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:48068
I20260926 04:07:40.445012 2502 tablet_replica.cc:330] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: stopping tablet replica
I20260926 04:07:40.445016 2500 tablet_replica.cc:330] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: stopping tablet replica
I20260926 04:07:40.445040 2501 tablet_replica.cc:330] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: stopping tablet replica
I20260926 04:07:40.445058 2502 raft_consensus.cc:2229] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:40.445086 2500 raft_consensus.cc:2229] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:40.445091 2501 raft_consensus.cc:2229] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:40.445116 2502 raft_consensus.cc:2258] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:40.445125 2501 raft_consensus.cc:2258] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:40.445127 2500 raft_consensus.cc:2258] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:40.445461 2502 ts_tablet_manager.cc:1905] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:40.445489 2500 ts_tablet_manager.cc:1905] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:40.446291 2501 ts_tablet_manager.cc:1905] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:40.447248 2500 ts_tablet_manager.cc:1918] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:40.447256 2502 ts_tablet_manager.cc:1918] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:40.447304 2500 log.cc:1196] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/b562ef7a08384a18a07c2890851e8058
I20260926 04:07:40.447329 2502 log.cc:1196] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/b562ef7a08384a18a07c2890851e8058
I20260926 04:07:40.447544 2500 ts_tablet_manager.cc:1939] T b562ef7a08384a18a07c2890851e8058 P 9f7cda66c96b4bb892e70871c47912f8: Deleting consensus metadata
I20260926 04:07:40.447561 2502 ts_tablet_manager.cc:1939] T b562ef7a08384a18a07c2890851e8058 P 5accc736c00a4184ab60b1aeb8d6676d: Deleting consensus metadata
I20260926 04:07:40.447890 1957 catalog_manager.cc:4841] TS 9f7cda66c96b4bb892e70871c47912f8 (127.0.213.195:44079): tablet b562ef7a08384a18a07c2890851e8058 (table test-table [id=9f43694f36cd459d94467b6e8133e567]) successfully deleted
I20260926 04:07:40.447959 1957 catalog_manager.cc:4841] TS 5accc736c00a4184ab60b1aeb8d6676d (127.0.213.193:42057): tablet b562ef7a08384a18a07c2890851e8058 (table test-table [id=9f43694f36cd459d94467b6e8133e567]) successfully deleted
I20260926 04:07:40.447983 2501 ts_tablet_manager.cc:1918] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:40.448040 2501 log.cc:1196] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/b562ef7a08384a18a07c2890851e8058
I20260926 04:07:40.448287 2501 ts_tablet_manager.cc:1939] T b562ef7a08384a18a07c2890851e8058 P f2858713773c4b51b12205821bc87914: Deleting consensus metadata
I20260926 04:07:40.448771 1957 catalog_manager.cc:4841] TS f2858713773c4b51b12205821bc87914 (127.0.213.194:38689): tablet b562ef7a08384a18a07c2890851e8058 (table test-table [id=9f43694f36cd459d94467b6e8133e567]) successfully deleted
Sep 26 04:07:40 dist-test-slave-xf24 krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:40.458729 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.448646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.0.213.193:42057)
0926 04:07:40.448719 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.448722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.448838 (+ 116us) server_negotiation.cc:407] Connection header received
0926 04:07:40.448909 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.448910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.448925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.448981 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:40.449501 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.450326 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.451100 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.451434 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.451514 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.454734 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:40.454746 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:40.454746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:40.454755 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:40.456930 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:40.457335 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:40.457339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:40.457340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:40.457410 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:40.458056 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:40.458056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:40.458057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:40.458252 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:40.458411 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.458523 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.458617 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 04:07:40.459196 2167 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48710
I20260926 04:07:40.467265 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:40.460436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.213.254:46795)
0926 04:07:40.460499 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:40.460501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:40.460511 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:40.460566 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:40.460567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:40.460578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:40.460639 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:40.461200 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.461754 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.462550 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:40.462830 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:40.462905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:40.464477 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:40.464478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:40.464479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:40.464485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:40.465978 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:40.466432 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:40.466433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:40.466433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:40.466492 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:40.466739 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:40.466741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:40.466741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:40.466864 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:40.466960 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:40.467117 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 04:07:40.467182 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 04:07:40.467583 1969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46380
I20260926 04:07:40.468242 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2042
I20260926 04:07:40.477097 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2182
I20260926 04:07:40.482726 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2322
I20260926 04:07:40.487787 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 1938
2026-09-26T04:07:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2480 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:40 dist-test-slave-xf24 krb5kdc[2540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:40 dist-test-slave-xf24 krb5kdc[2540](info): set up 2 sockets
Sep 26 04:07:40 dist-test-slave-xf24 krb5kdc[2540](info): commencing operation
krb5kdc: starting...
W20260926 04:07:41.527729 855 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:41 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:41.647878 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:42851
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:41.728426 2553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:41.728632 2553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:41.728662 2553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:41.729833 2553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:41.729875 2553 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 04:07:41.729888 2553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:41.729902 2553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:41.729914 2553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:41.729925 2553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:41.731500 2553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:42851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:41.731801 2553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:41.732045 2553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:41.737287 2561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:41.737324 2559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:41.737506 2558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:41.737562 2553 server_base.cc:1034] running on GCE node
I20260926 04:07:41.737826 2553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:41.738127 2553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:41.739302 2553 hybrid_clock.cc:648] HybridClock initialized: now 1790395661739315 us; error 58 us; skew 500 ppm
Sep 26 04:07:41 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:41.741928 2553 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:41.742446 2553 webserver.cc:460] Webserver started at http://127.0.213.254:46655/ using document root <none> and password file <none>
I20260926 04:07:41.742687 2553 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:41.742756 2553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:41.742933 2553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:41.744027 2553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "ebcb63327c7b466bbaeec3a176df65e2"
format_stamp: "Formatted at 2026-09-26 04:07:41 on dist-test-slave-xf24"
server_key: "067e0f8e3a91e5e8981364f55ec190e1"
server_key_iv: "f0cf600c04a4939d7e4cddb484d7f855"
server_key_version: "encryptionkey@0"
I20260926 04:07:41.744460 2553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "ebcb63327c7b466bbaeec3a176df65e2"
format_stamp: "Formatted at 2026-09-26 04:07:41 on dist-test-slave-xf24"
server_key: "067e0f8e3a91e5e8981364f55ec190e1"
server_key_iv: "f0cf600c04a4939d7e4cddb484d7f855"
server_key_version: "encryptionkey@0"
I20260926 04:07:41.745862 2553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:41.746627 2568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:41.746850 2553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:41.746958 2553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "ebcb63327c7b466bbaeec3a176df65e2"
format_stamp: "Formatted at 2026-09-26 04:07:41 on dist-test-slave-xf24"
server_key: "067e0f8e3a91e5e8981364f55ec190e1"
server_key_iv: "f0cf600c04a4939d7e4cddb484d7f855"
server_key_version: "encryptionkey@0"
I20260926 04:07:41.747053 2553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:41.755887 2553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:41.757741 2553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:41.757905 2553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:41.762603 2553 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:42851
I20260926 04:07:41.762686 2630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:42851 every 8 connection(s)
I20260926 04:07:41.763039 2553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:41.763343 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2553
I20260926 04:07:41.763437 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:41.763615 855 external_mini_cluster.cc:1428] Setting key 2c5425a410bbcfc2b2394edf74ebbacb
I20260926 04:07:41.763854 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:41.767186 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Bootstrap starting.
I20260926 04:07:41.767809 2631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:41.768201 2631 log.cc:824] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:41.768878 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: No bootstrap required, opened a new log
Sep 26 04:07:41 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:41.771183 2631 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb63327c7b466bbaeec3a176df65e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42851 } }
I20260926 04:07:41.771353 2631 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:41.771399 2631 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcb63327c7b466bbaeec3a176df65e2, State: Initialized, Role: FOLLOWER
I20260926 04:07:41.771530 2631 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb63327c7b466bbaeec3a176df65e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42851 } }
I20260926 04:07:41.771626 2631 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:41.771665 2631 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:41.771706 2631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:41.772359 2631 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb63327c7b466bbaeec3a176df65e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42851 } }
I20260926 04:07:41.772480 2631 leader_election.cc:302] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebcb63327c7b466bbaeec3a176df65e2; no voters:
I20260926 04:07:41.772687 2631 leader_election.cc:288] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:41.772753 2636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:41.772897 2636 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [term 1 LEADER]: Becoming Leader. State: Replica: ebcb63327c7b466bbaeec3a176df65e2, State: Running, Role: LEADER
I20260926 04:07:41.772972 2631 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:41.772991 2636 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb63327c7b466bbaeec3a176df65e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42851 } }
I20260926 04:07:41.773877 2638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebcb63327c7b466bbaeec3a176df65e2. Latest consensus state: current_term: 1 leader_uuid: "ebcb63327c7b466bbaeec3a176df65e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb63327c7b466bbaeec3a176df65e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42851 } } }
I20260926 04:07:41.773993 2638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:41.774471 2637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebcb63327c7b466bbaeec3a176df65e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebcb63327c7b466bbaeec3a176df65e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42851 } } }
I20260926 04:07:41.774556 2637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:41.774873 2641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:41.775513 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:41.764195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.0.213.254:42851)
0926 04:07:41.764396 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:41.764407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:41.764439 (+ 32us) server_negotiation.cc:407] Connection header received
0926 04:07:41.765036 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:41.765045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:41.765107 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:41.765259 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:41.765869 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.767198 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.768023 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.768543 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.768681 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:41.771360 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:41.771365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:41.771365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:41.771379 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:41.773516 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:41.773942 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:41.773945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:41.773945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:41.774052 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:41.774274 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:41.774275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:41.774275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:41.774619 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:41.774784 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:41.775101 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 04:07:41.775265 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260926 04:07:41.775372 2641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:41.777369 2641 catalog_manager.cc:1347] Generated new cluster ID: f8555431acaa4b0d8c3ceb3380e8217b
I20260926 04:07:41.777441 2641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:41.787686 2641 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:41.788302 2641 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:41.805311 2641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Generated new TSK 0
I20260926 04:07:41.805533 2641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:41.843209 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:42851
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:41.926074 2657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:41.926316 2657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:41.926357 2657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:41.927531 2657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:41.927605 2657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:41.927639 2657 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 04:07:41.927670 2657 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:07:41.927711 2657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:41.929217 2657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:41.929497 2657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:41.929744 2657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:41.934928 2662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:41.934934 2665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:41.934934 2663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:41.935104 2657 server_base.cc:1034] running on GCE node
I20260926 04:07:41.935359 2657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:41.935607 2657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:41.936779 2657 hybrid_clock.cc:648] HybridClock initialized: now 1790395661936754 us; error 27 us; skew 500 ppm
Sep 26 04:07:41 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:41.939265 2657 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:41.939678 2657 webserver.cc:460] Webserver started at http://127.0.213.193:33887/ using document root <none> and password file <none>
I20260926 04:07:41.939872 2657 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:41.939914 2657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:41.940003 2657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:41.941083 2657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a"
format_stamp: "Formatted at 2026-09-26 04:07:41 on dist-test-slave-xf24"
server_key: "c101aa713d40168715a8edeaf590ac8a"
server_key_iv: "5ba279c4e584245454792ae54466f015"
server_key_version: "encryptionkey@0"
I20260926 04:07:41.941406 2657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a"
format_stamp: "Formatted at 2026-09-26 04:07:41 on dist-test-slave-xf24"
server_key: "c101aa713d40168715a8edeaf590ac8a"
server_key_iv: "5ba279c4e584245454792ae54466f015"
server_key_version: "encryptionkey@0"
I20260926 04:07:41.942709 2657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:41.943514 2672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:41.943715 2657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:41.943799 2657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a"
format_stamp: "Formatted at 2026-09-26 04:07:41 on dist-test-slave-xf24"
server_key: "c101aa713d40168715a8edeaf590ac8a"
server_key_iv: "5ba279c4e584245454792ae54466f015"
server_key_version: "encryptionkey@0"
I20260926 04:07:41.943856 2657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:41.957269 2657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:41.959633 2657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:41.959753 2657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:41.960309 2657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:41.960350 2657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:41.960372 2657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:41.960390 2657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:41.967569 2657 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:36869
I20260926 04:07:41.967687 2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:36869 every 8 connection(s)
I20260926 04:07:41.967988 2657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
Sep 26 04:07:41 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:41.968449 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2657
I20260926 04:07:41.968541 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:41.968734 855 external_mini_cluster.cc:1428] Setting key eb2b805b176a3cad3f82c7c0dfba86a0
I20260926 04:07:41.972150 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:41.960977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:56671 (local address 127.0.213.254:42851)
0926 04:07:41.961038 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:41.961040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:41.961302 (+ 262us) server_negotiation.cc:407] Connection header received
0926 04:07:41.961955 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:41.961957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:41.961970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:41.962048 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:41.963015 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.963991 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.965723 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.965945 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.966007 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:41.968974 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:41.968975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:41.968976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:41.968984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:41.970497 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:41.971025 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:41.971027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:41.971027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:41.971107 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:41.971368 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:41.971369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:41.971370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:41.971562 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:41.971646 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:41.971941 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 04:07:41.972034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:07:41.972384 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:41.960943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.213.193:56671)
0926 04:07:41.961173 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:41.961212 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:41.961850 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:41.962177 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:41.962183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:41.962586 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:41.962914 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:41.962922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.964131 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.964132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:41.965614 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:41.965617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.966043 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.966044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:41.966216 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:41.966910 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:41.966922 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:41.968809 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:41.970589 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:41.970596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:41.970598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:41.970924 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:41.971186 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:41.971187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:41.971187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:41.971297 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:41.971639 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:41.971643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:41.971769 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 04:07:41.971890 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 04:07:41.972056 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260926 04:07:41.972940 2788 heartbeater.cc:347] Connected to a master server at 127.0.213.254:42851
I20260926 04:07:41.973037 2788 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:41.973243 2788 heartbeater.cc:511] Master 127.0.213.254:42851 requested a full tablet report, sending...
I20260926 04:07:41.973855 2584 ts_manager.cc:194] Registered new tserver with Master: 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869)
I20260926 04:07:41.974947 2584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:56671
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:07:41.980581 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:41.974216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.213.193:41187)
0926 04:07:41.974384 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:41.974397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:41.974477 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:41.974656 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:41.974657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:41.974901 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:41.974979 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:41.974981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.975864 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.975864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:41.976455 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:41.976457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.976985 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.976985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:41.977038 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:41.977558 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:41.977565 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:41.977842 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:41.979562 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:41.979563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:41.979563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:41.979723 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:41.979979 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:41.979980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:41.979981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:41.980055 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:41.980324 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:41.980325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:41.980382 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 04:07:41.980437 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:07:41.980514 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":119,"threads_started":1}
I20260926 04:07:41.980588 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:41.974218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:41187 (local address 127.0.213.254:42851)
0926 04:07:41.974253 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:41.974255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:41.974450 (+ 195us) server_negotiation.cc:407] Connection header received
0926 04:07:41.974546 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:41.974547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:41.974557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:41.974588 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:41.975082 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.975749 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.976545 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:41.976898 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:41.976984 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:41.977935 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:41.977936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:41.977937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:41.977945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:41.979490 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:41.979791 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:41.979791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:41.979792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:41.979840 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:41.980150 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:41.980153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:41.980153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:41.980262 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:41.980319 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:41.980467 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 04:07:41.980526 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:42.000111 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:42851
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:42.082216 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:42.082427 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:42.082451 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:42.083554 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:42.083590 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:42.083609 2797 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 04:07:42.083621 2797 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:07:42.083640 2797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:42.085072 2797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:42.085256 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:42.085430 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:42.091392 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:42.091353 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:42.091439 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:42.091599 2797 server_base.cc:1034] running on GCE node
I20260926 04:07:42.091784 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:42.092041 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:42.093287 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790395662093271 us; error 32 us; skew 500 ppm
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:42.096035 2797 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:42.096592 2797 webserver.cc:460] Webserver started at http://127.0.213.194:40067/ using document root <none> and password file <none>
I20260926 04:07:42.096850 2797 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:42.096921 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:42.097077 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:42.098191 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "dce473b2bc8043e58b7f959e7ee35536"
format_stamp: "Formatted at 2026-09-26 04:07:42 on dist-test-slave-xf24"
server_key: "783fc08e5123a9a42e3e3b41426bfeb2"
server_key_iv: "e44bc6d8f89f9ee6fbf4caef3ec491e6"
server_key_version: "encryptionkey@0"
I20260926 04:07:42.098569 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "dce473b2bc8043e58b7f959e7ee35536"
format_stamp: "Formatted at 2026-09-26 04:07:42 on dist-test-slave-xf24"
server_key: "783fc08e5123a9a42e3e3b41426bfeb2"
server_key_iv: "e44bc6d8f89f9ee6fbf4caef3ec491e6"
server_key_version: "encryptionkey@0"
I20260926 04:07:42.100013 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:07:42.100847 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:42.101064 2797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:42.101166 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "dce473b2bc8043e58b7f959e7ee35536"
format_stamp: "Formatted at 2026-09-26 04:07:42 on dist-test-slave-xf24"
server_key: "783fc08e5123a9a42e3e3b41426bfeb2"
server_key_iv: "e44bc6d8f89f9ee6fbf4caef3ec491e6"
server_key_version: "encryptionkey@0"
I20260926 04:07:42.101263 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:42.140537 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:42.142951 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:42.143170 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:42.143815 2797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:42.143870 2797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:42.143908 2797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:42.143961 2797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:42.151867 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:35345
I20260926 04:07:42.151947 2927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:35345 every 8 connection(s)
I20260926 04:07:42.152346 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:42.154969 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.144338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:40891 (local address 127.0.213.254:42851)
0926 04:07:42.144413 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.144416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.144724 (+ 308us) server_negotiation.cc:407] Connection header received
0926 04:07:42.145483 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.145484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.145494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.145538 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:42.146523 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.147356 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.149140 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.149489 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.149570 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.152078 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:42.152081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:42.152081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:42.152090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:42.153673 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.154123 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.154124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.154124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.154181 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.154397 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.154398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.154398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.154507 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:42.154560 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.154823 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.154901 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:07:42.155115 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.144329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.213.194:40891)
0926 04:07:42.144557 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.144625 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.145369 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.145666 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.145673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.146097 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:42.146408 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.146416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.147464 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.147465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.149035 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.149039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.149600 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.149602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.149747 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.150329 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:42.150348 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:42.151961 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:42.153796 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.153802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.153804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.154052 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.154242 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.154242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.154242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.154342 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.154564 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:42.154568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:42.154705 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.154792 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.154922 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
I20260926 04:07:42.155526 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2797
I20260926 04:07:42.155619 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:42.155692 2928 heartbeater.cc:347] Connected to a master server at 127.0.213.254:42851
I20260926 04:07:42.155777 2928 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:42.155821 855 external_mini_cluster.cc:1428] Setting key 5215eaa47b09838e0414116b6841d498
I20260926 04:07:42.155972 2928 heartbeater.cc:511] Master 127.0.213.254:42851 requested a full tablet report, sending...
I20260926 04:07:42.156589 2585 ts_manager.cc:194] Registered new tserver with Master: dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345)
I20260926 04:07:42.157111 2585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:40891
I20260926 04:07:42.164378 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.156914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:45225 (local address 127.0.213.254:42851)
0926 04:07:42.157936 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.157937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.157944 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:07:42.157960 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.157960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.157968 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.158046 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:42.158403 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.158880 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.159980 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.160324 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.160421 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.161553 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:42.161554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:42.161554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:42.161561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:42.163143 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.163548 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.163549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.163549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.163601 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.163811 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.163812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.163812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.163953 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:42.164009 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.164241 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.164307 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003}
I20260926 04:07:42.164448 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.156752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.213.194:45225)
0926 04:07:42.156932 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.156942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.156987 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.158054 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.158055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.158260 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:42.158312 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.158313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.158979 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.158980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.159887 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.159890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.160435 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.160436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.160527 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.161006 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:42.161011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:42.161458 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:42.163242 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.163244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.163244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.163468 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.163671 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.163672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.163673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.163741 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.164028 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:42.164029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:42.164124 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.164219 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.164344 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:42.182857 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:42851
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:42.263180 2937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:42.263391 2937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:42.263417 2937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:42.264551 2937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:42.264590 2937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:42.264609 2937 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 04:07:42.264621 2937 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:07:42.264638 2937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:42.266022 2937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:42.266213 2937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:42.266393 2937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:42.271656 2945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:42.271775 2937 server_base.cc:1034] running on GCE node
W20260926 04:07:42.271656 2942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:42.271682 2943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:42.272006 2937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:42.272322 2937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:42.273486 2937 hybrid_clock.cc:648] HybridClock initialized: now 1790395662273468 us; error 38 us; skew 500 ppm
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:42.277104 2937 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:42.277647 2937 webserver.cc:460] Webserver started at http://127.0.213.195:45649/ using document root <none> and password file <none>
I20260926 04:07:42.277909 2937 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:42.277978 2937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:42.278141 2937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:42.279243 2937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "315cbb9bb61e493fac27d2d13300b6ea"
format_stamp: "Formatted at 2026-09-26 04:07:42 on dist-test-slave-xf24"
server_key: "04fdc39730a8e5ff1bcb481008ac97f3"
server_key_iv: "203090b19714f2d1754913eef1e85801"
server_key_version: "encryptionkey@0"
I20260926 04:07:42.279619 2937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "315cbb9bb61e493fac27d2d13300b6ea"
format_stamp: "Formatted at 2026-09-26 04:07:42 on dist-test-slave-xf24"
server_key: "04fdc39730a8e5ff1bcb481008ac97f3"
server_key_iv: "203090b19714f2d1754913eef1e85801"
server_key_version: "encryptionkey@0"
I20260926 04:07:42.281064 2937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:42.281898 2952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:42.282119 2937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:42.282223 2937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "315cbb9bb61e493fac27d2d13300b6ea"
format_stamp: "Formatted at 2026-09-26 04:07:42 on dist-test-slave-xf24"
server_key: "04fdc39730a8e5ff1bcb481008ac97f3"
server_key_iv: "203090b19714f2d1754913eef1e85801"
server_key_version: "encryptionkey@0"
I20260926 04:07:42.282296 2937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:42.294595 2937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:42.297164 2937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:42.297360 2937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:42.297991 2937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:42.298059 2937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:42.298094 2937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:42.298122 2937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:42.305567 2937 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:42669
I20260926 04:07:42.305667 3067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:42669 every 8 connection(s)
I20260926 04:07:42.305965 2937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:42.307680 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 2937
I20260926 04:07:42.307770 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:42.307974 855 external_mini_cluster.cc:1428] Setting key 2ed7e9bd1a82cfd531e1623a2286bdd9
I20260926 04:07:42.309008 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.298377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34959 (local address 127.0.213.254:42851)
0926 04:07:42.298436 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.298439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.298704 (+ 265us) server_negotiation.cc:407] Connection header received
0926 04:07:42.299320 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.299322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.299334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.299386 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:42.300305 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.300864 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.301893 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.302140 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.302227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.305638 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:42.305639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:42.305639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:42.305648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:42.307324 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.307826 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.307828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.307828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.307892 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.308272 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.308275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.308275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.308432 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:42.308499 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.308827 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.308908 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 04:07:42.309270 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.298364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.213.195:34959)
0926 04:07:42.298574 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.298612 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.299200 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.299465 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.299469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.299881 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:42.300202 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.300208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.300962 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.300963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.301785 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.301787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.302238 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.302239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.302372 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.302969 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:42.302979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:42.305546 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:42.307408 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.307413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.307415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.307737 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.307973 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.307974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.307974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.308179 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.308505 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:42.308511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:42.308673 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.308770 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.308963 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":135,"threads_started":1}
I20260926 04:07:42.309715 3068 heartbeater.cc:347] Connected to a master server at 127.0.213.254:42851
I20260926 04:07:42.309799 3068 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:42.309998 3068 heartbeater.cc:511] Master 127.0.213.254:42851 requested a full tablet report, sending...
I20260926 04:07:42.310307 2585 ts_manager.cc:194] Registered new tserver with Master: 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:42.310653 2585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:34959
I20260926 04:07:42.316004 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.310764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:53139 (local address 127.0.213.254:42851)
0926 04:07:42.310966 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.310968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.310976 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:42.310995 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.310996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.311005 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.311072 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:42.311389 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.311896 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.312655 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.312833 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.312888 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.313808 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:42.313809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:42.313809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:42.313815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:42.315111 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.315422 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.315423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.315423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.315475 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.315637 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.315637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.315637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.315727 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:42.315774 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.315885 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.315940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260926 04:07:42.316017 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.310715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.213.195:53139)
0926 04:07:42.310836 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.310846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.310895 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.311075 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.311076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.311261 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:42.311316 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.311317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.311983 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.311984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.312576 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.312578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.312898 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.312899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.312949 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.313416 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:42.313421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:42.313727 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:42.315192 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.315193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.315194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.315352 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.315532 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.315533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.315533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.315581 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.315779 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:42.315779 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:42.315816 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.315866 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.315943 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260926 04:07:42.319821 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:42.344008 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.336236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.0.213.254:42851)
0926 04:07:42.336320 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.336322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.336332 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:42.336357 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.336358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.336368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.336461 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:42.337131 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.337636 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.338391 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.338586 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.338689 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.341045 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:42.341047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:42.341047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:42.341056 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:42.342496 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.342894 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.342895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.342895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.343010 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.343405 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.343406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.343407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.343565 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:42.343674 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.343819 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.343907 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:07:42.345836 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:42.347723 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:42.354638 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.350554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.0.213.193:36869)
0926 04:07:42.350894 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.350899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.350909 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:42.350945 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.350947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.350963 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.351229 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.351288 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.352429 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.353598 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.354196 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.354300 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.354364 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.354481 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.354519 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":76,"threads_started":1}
I20260926 04:07:42.354638 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.350415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36869 (local address 127.0.0.1:40558)
0926 04:07:42.350590 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.350634 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.350701 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.351071 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.351075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.351077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.351150 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.351152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.352548 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.352549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.353514 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.353516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.354302 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.354303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.354379 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.354393 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.354466 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.354526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":57,"threads_started":1}
I20260926 04:07:42.354864 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.350419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42669 (local address 127.0.0.1:42014)
0926 04:07:42.350709 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.350718 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.350760 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.351148 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.351149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.351151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.351220 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.351222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.352527 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.352528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.353683 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.353685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.354627 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.354627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.354685 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.354688 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.354737 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.354779 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":56,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260926 04:07:42.355172 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.350680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.0.213.195:42669)
0926 04:07:42.350883 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.350888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.350896 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:42.350936 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.350938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.350955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.351080 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.351289 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.352429 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.353766 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.354482 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.354548 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.354595 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.355033 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.355074 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260926 04:07:42.355450 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.350678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.0.213.194:35345)
0926 04:07:42.350857 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.350863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.350878 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:07:42.350941 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.350943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.350956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.351043 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.351316 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.352017 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.353502 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.354239 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.354312 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.354365 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.354675 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.355094 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260926 04:07:42.355415 2722 tablet_service.cc:1467] Processing CreateTablet for tablet a98e428bc4cb4901b8efd66a0cbc9728 (DEFAULT_TABLE table=test-table [id=4c12bbefdac7444880d8a4a10d47467f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:42.355849 2722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e428bc4cb4901b8efd66a0cbc9728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.355784 3002 tablet_service.cc:1467] Processing CreateTablet for tablet a98e428bc4cb4901b8efd66a0cbc9728 (DEFAULT_TABLE table=test-table [id=4c12bbefdac7444880d8a4a10d47467f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:42.356033 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e428bc4cb4901b8efd66a0cbc9728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.356284 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.350419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:35345 (local address 127.0.0.1:41332)
0926 04:07:42.350810 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.350818 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.350855 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.351154 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.351154 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.351155 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.351242 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.351244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.352162 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.352163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.353405 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.353408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.354401 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.354402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.354488 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.354494 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.354541 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.354587 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":38,"thread_start_us":126,"threads_started":1}
I20260926 04:07:42.357182 2862 tablet_service.cc:1467] Processing CreateTablet for tablet a98e428bc4cb4901b8efd66a0cbc9728 (DEFAULT_TABLE table=test-table [id=4c12bbefdac7444880d8a4a10d47467f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:42.357544 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e428bc4cb4901b8efd66a0cbc9728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.359234 3091 tablet_bootstrap.cc:492] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Bootstrap starting.
I20260926 04:07:42.359849 3091 tablet_bootstrap.cc:654] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.360217 3091 log.cc:824] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:42.360970 3091 tablet_bootstrap.cc:492] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: No bootstrap required, opened a new log
I20260926 04:07:42.361070 3091 ts_tablet_manager.cc:1397] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:42.361121 3094 tablet_bootstrap.cc:492] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Bootstrap starting.
I20260926 04:07:42.361503 3092 tablet_bootstrap.cc:492] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Bootstrap starting.
I20260926 04:07:42.361745 3094 tablet_bootstrap.cc:654] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.362133 3094 log.cc:824] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:42.362388 3092 tablet_bootstrap.cc:654] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.362766 3092 log.cc:824] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:42.362926 3094 tablet_bootstrap.cc:492] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: No bootstrap required, opened a new log
I20260926 04:07:42.363018 3094 ts_tablet_manager.cc:1397] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:42.363044 3091 raft_consensus.cc:348] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.363240 3091 raft_consensus.cc:374] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.363276 3091 raft_consensus.cc:729] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.363417 3091 consensus_queue.cc:260] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.363668 3092 tablet_bootstrap.cc:492] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: No bootstrap required, opened a new log
I20260926 04:07:42.363674 3091 ts_tablet_manager.cc:1428] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:42.363689 2788 heartbeater.cc:503] Master 127.0.213.254:42851 was elected leader, sending a full tablet report...
I20260926 04:07:42.363739 3092 ts_tablet_manager.cc:1397] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:42.365020 3094 raft_consensus.cc:348] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.365193 3094 raft_consensus.cc:374] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.365226 3094 raft_consensus.cc:729] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce473b2bc8043e58b7f959e7ee35536, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.365338 3094 consensus_queue.cc:260] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.365612 3094 ts_tablet_manager.cc:1428] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:42.365664 2928 heartbeater.cc:503] Master 127.0.213.254:42851 was elected leader, sending a full tablet report...
I20260926 04:07:42.365775 3092 raft_consensus.cc:348] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.365883 3092 raft_consensus.cc:374] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.365904 3092 raft_consensus.cc:729] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315cbb9bb61e493fac27d2d13300b6ea, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.366005 3092 consensus_queue.cc:260] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.366261 3092 ts_tablet_manager.cc:1428] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:07:42.366293 3068 heartbeater.cc:503] Master 127.0.213.254:42851 was elected leader, sending a full tablet report...
I20260926 04:07:42.371629 3098 raft_consensus.cc:482] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:42.371718 3098 raft_consensus.cc:504] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.371982 3098 leader_election.cc:288] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669), 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869)
I20260926 04:07:42.375399 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.372223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36869 (local address 127.0.213.194:43033)
0926 04:07:42.372385 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.372394 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.372437 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.372562 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.372563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.372564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.372636 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.372638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.373436 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.373437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.374603 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.374605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.375141 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.375142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.375227 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.375231 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.375283 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.375319 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":128,"thread_start_us":93,"threads_started":1}
I20260926 04:07:42.375417 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.372271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:43033 (local address 127.0.213.193:36869)
0926 04:07:42.372312 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.372315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.372446 (+ 131us) server_negotiation.cc:407] Connection header received
0926 04:07:42.372498 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.372499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.372509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.372558 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.372722 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.373330 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.374688 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.375067 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.375122 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.375129 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.375298 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.375340 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:07:42.375499 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.372223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42669 (local address 127.0.213.194:42185)
0926 04:07:42.372382 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.372391 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.372453 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.372655 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.372656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.372657 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.372719 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.372720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.373415 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.373416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.374607 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.374609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.375228 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.375231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.375324 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.375328 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.375384 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.375418 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":49}
I20260926 04:07:42.375510 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.372250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:42185 (local address 127.0.213.195:42669)
0926 04:07:42.372514 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.372516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.372524 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:42.372555 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.372555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.372564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.372615 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.372811 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.373331 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.374690 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.375146 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.375216 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.375224 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.375398 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.375440 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260926 04:07:42.375749 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e428bc4cb4901b8efd66a0cbc9728" candidate_uuid: "dce473b2bc8043e58b7f959e7ee35536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea" is_pre_election: true
I20260926 04:07:42.375751 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e428bc4cb4901b8efd66a0cbc9728" candidate_uuid: "dce473b2bc8043e58b7f959e7ee35536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" is_pre_election: true
I20260926 04:07:42.375893 2742 raft_consensus.cc:2454] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dce473b2bc8043e58b7f959e7ee35536 in term 0.
I20260926 04:07:42.375893 3022 raft_consensus.cc:2454] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dce473b2bc8043e58b7f959e7ee35536 in term 0.
I20260926 04:07:42.376163 2816 leader_election.cc:302] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:42.376327 3098 raft_consensus.cc:2781] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:42.376370 3098 raft_consensus.cc:482] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:42.376386 3098 raft_consensus.cc:3037] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.377045 3098 raft_consensus.cc:504] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.377182 3098 leader_election.cc:288] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [CANDIDATE]: Term 1 election: Requested vote from peers 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669), 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869)
I20260926 04:07:42.377321 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e428bc4cb4901b8efd66a0cbc9728" candidate_uuid: "dce473b2bc8043e58b7f959e7ee35536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea"
I20260926 04:07:42.377338 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e428bc4cb4901b8efd66a0cbc9728" candidate_uuid: "dce473b2bc8043e58b7f959e7ee35536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a"
I20260926 04:07:42.377403 3022 raft_consensus.cc:3037] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.377403 2742 raft_consensus.cc:3037] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.378235 2742 raft_consensus.cc:2454] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dce473b2bc8043e58b7f959e7ee35536 in term 1.
I20260926 04:07:42.378235 3022 raft_consensus.cc:2454] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dce473b2bc8043e58b7f959e7ee35536 in term 1.
I20260926 04:07:42.378731 2813 leader_election.cc:302] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315cbb9bb61e493fac27d2d13300b6ea, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:42.378914 3098 raft_consensus.cc:2781] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:42.379148 3098 raft_consensus.cc:686] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 1 LEADER]: Becoming Leader. State: Replica: dce473b2bc8043e58b7f959e7ee35536, State: Running, Role: LEADER
I20260926 04:07:42.379256 3098 consensus_queue.cc:237] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } }
I20260926 04:07:42.380030 2585 catalog_manager.cc:5495] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 reported cstate change: term changed from 0 to 1, leader changed from <none> to dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194). New cstate: current_term: 1 leader_uuid: "dce473b2bc8043e58b7f959e7ee35536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } health_report { overall_health: UNKNOWN } } }
W20260926 04:07:42.403312 2929 tablet.cc:2369] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 04:07:42 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:42.416570 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.408808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42851 (local address 127.0.0.1:37484)
0926 04:07:42.409024 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.409037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.409113 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.409247 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.409247 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.409618 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:42.409674 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.409676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.410524 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.410525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.411078 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.411080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.411455 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.411456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.411523 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.412149 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:42.412154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:42.413703 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:42.415396 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.415404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.415406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.415616 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.415862 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:42.415863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:42.415863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:42.415928 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:42.416255 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:42.416261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:42.416306 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.416366 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.416464 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":157,"threads_started":1}
I20260926 04:07:42.416616 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.408875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.0.213.254:42851)
0926 04:07:42.408957 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.408960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.409107 (+ 147us) server_negotiation.cc:407] Connection header received
0926 04:07:42.409159 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.409160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.409169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.409232 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:42.409793 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.410410 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.411177 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.411362 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.411425 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.413802 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:42.413803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:42.413804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:42.413810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:42.415297 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.415730 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.415731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.415732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.415789 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:42.416005 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:42.416006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:42.416006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:42.416194 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:42.416252 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.416397 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.416512 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":34}
I20260926 04:07:42.418686 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:37484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 04:07:42.418874 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:42.420614 2722 tablet_service.cc:1467] Processing CreateTablet for tablet 68214f85d8ba499b8186bf10d53a2c38 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb73595e5e6147d99029ea931a53014d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:07:42.420612 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 68214f85d8ba499b8186bf10d53a2c38 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb73595e5e6147d99029ea931a53014d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:07:42.420795 2722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68214f85d8ba499b8186bf10d53a2c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.420806 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68214f85d8ba499b8186bf10d53a2c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.420902 3002 tablet_service.cc:1467] Processing CreateTablet for tablet 68214f85d8ba499b8186bf10d53a2c38 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb73595e5e6147d99029ea931a53014d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:07:42.421038 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68214f85d8ba499b8186bf10d53a2c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.422051 3094 tablet_bootstrap.cc:492] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: Bootstrap starting.
I20260926 04:07:42.422354 3091 tablet_bootstrap.cc:492] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Bootstrap starting.
I20260926 04:07:42.422488 3094 tablet_bootstrap.cc:654] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.422670 3092 tablet_bootstrap.cc:492] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: Bootstrap starting.
I20260926 04:07:42.423121 3091 tablet_bootstrap.cc:654] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.423188 3094 tablet_bootstrap.cc:492] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: No bootstrap required, opened a new log
I20260926 04:07:42.423235 3094 ts_tablet_manager.cc:1397] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:42.423249 3092 tablet_bootstrap.cc:654] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.423393 3094 raft_consensus.cc:348] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.423470 3094 raft_consensus.cc:374] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.423506 3094 raft_consensus.cc:729] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce473b2bc8043e58b7f959e7ee35536, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.423560 3094 consensus_queue.cc:260] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.423691 3094 ts_tablet_manager.cc:1428] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:42.423851 3091 tablet_bootstrap.cc:492] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: No bootstrap required, opened a new log
I20260926 04:07:42.423895 3091 ts_tablet_manager.cc:1397] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:42.423899 3092 tablet_bootstrap.cc:492] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: No bootstrap required, opened a new log
I20260926 04:07:42.423943 3092 ts_tablet_manager.cc:1397] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:42.423933 3098 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.424029 3098 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.424033 3091 raft_consensus.cc:348] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.424144 3091 raft_consensus.cc:374] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.424111 3092 raft_consensus.cc:348] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.424167 3091 raft_consensus.cc:729] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.424189 3092 raft_consensus.cc:374] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.424216 3092 raft_consensus.cc:729] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315cbb9bb61e493fac27d2d13300b6ea, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.424214 3091 consensus_queue.cc:260] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.424258 3092 consensus_queue.cc:260] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.424306 3091 ts_tablet_manager.cc:1428] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:42.424345 3092 ts_tablet_manager.cc:1428] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:42.424371 3097 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.424431 3097 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.424486 3099 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.424568 3099 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.462044 3097 raft_consensus.cc:482] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:42.462172 3097 raft_consensus.cc:504] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.462544 3097 leader_election.cc:288] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:42.466596 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.462830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:45143 (local address 127.0.213.195:42669)
0926 04:07:42.462908 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.462910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.463031 (+ 121us) server_negotiation.cc:407] Connection header received
0926 04:07:42.463103 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.463104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.463113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.463168 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.463319 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.464093 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.465587 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.466132 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.466216 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.466224 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.466467 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.466540 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 04:07:42.466586 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.462869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:42667 (local address 127.0.213.194:35345)
0926 04:07:42.462932 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.462934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.462942 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:42.462977 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.462978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.462986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.463108 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:42.463307 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.464050 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.465582 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.466150 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.466216 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.466224 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.466469 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.466517 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 04:07:42.466567 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.462772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:35345 (local address 127.0.213.193:42667)
0926 04:07:42.462831 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.462844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.462886 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.463124 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.463125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.463127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.463211 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.463213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.464165 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.464166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.465468 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.465472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.466241 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.466242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.466338 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.466341 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.466407 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.466463 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":42}
I20260926 04:07:42.466681 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.462772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42669 (local address 127.0.213.193:45143)
0926 04:07:42.462955 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.462964 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.463017 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.463181 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.463181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.463182 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:42.463241 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.463242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.464165 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.464166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.465468 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.465472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.466260 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.466261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.466336 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.466341 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.466407 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.466463 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":87,"thread_start_us":85,"threads_started":1}
I20260926 04:07:42.466960 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68214f85d8ba499b8186bf10d53a2c38" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536" is_pre_election: true
I20260926 04:07:42.467119 2882 raft_consensus.cc:2454] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:42.467098 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68214f85d8ba499b8186bf10d53a2c38" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea" is_pre_election: true
I20260926 04:07:42.467175 3022 raft_consensus.cc:2454] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:42.467331 2674 leader_election.cc:302] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:42.467518 3097 raft_consensus.cc:2781] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:42.467559 3097 raft_consensus.cc:482] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:42.467574 3097 raft_consensus.cc:3037] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.468428 3097 raft_consensus.cc:504] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.468575 3097 leader_election.cc:288] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Requested vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:42.468717 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68214f85d8ba499b8186bf10d53a2c38" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536"
W20260926 04:07:42.468765 2789 tablet.cc:2369] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:07:42.468804 2882 raft_consensus.cc:3037] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.468766 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68214f85d8ba499b8186bf10d53a2c38" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea"
I20260926 04:07:42.468853 3022 raft_consensus.cc:3037] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.469520 2882 raft_consensus.cc:2454] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:42.469501 3022 raft_consensus.cc:2454] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:42.469687 2674 leader_election.cc:302] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:42.469785 3097 raft_consensus.cc:2781] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:42.469945 3097 raft_consensus.cc:686] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Becoming Leader. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Running, Role: LEADER
I20260926 04:07:42.470050 3097 consensus_queue.cc:237] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.470554 3113 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: TxnStatusTablet state changed. Reason: New leader 17a4dc131fdf48df9947ba8ef3ae9b6a. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.470643 3113 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:07:42.470811 3115 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 04:07:42.470934 3115 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 04:07:42.470842 2585 catalog_manager.cc:5495] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193). New cstate: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:42.485486 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.482644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36869 (local address 127.0.0.1:40560)
0926 04:07:42.482830 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.482841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.482902 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.483068 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.483069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.483070 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:42.483124 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.483127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.483792 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.483794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.484454 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.484456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.484931 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.484932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.485007 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.485026 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.485313 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.485322 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.485367 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.485419 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 04:07:42.485484 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.482746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.0.213.193:36869)
0926 04:07:42.482798 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.482801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.482881 (+ 80us) server_negotiation.cc:407] Connection header received
0926 04:07:42.482991 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.482993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.483011 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.483067 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:42.483184 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.483693 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.484571 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.484847 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.484930 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.485132 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.485246 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.485335 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.485380 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.485401 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:07:42.487247 3022 raft_consensus.cc:1261] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:42.487282 2882 raft_consensus.cc:1261] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:42.487586 3097 consensus_queue.cc:1035] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:42.487752 3113 consensus_queue.cc:1035] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:42.488875 3098 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: TxnStatusTablet state changed. Reason: New leader 17a4dc131fdf48df9947ba8ef3ae9b6a. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.488974 3098 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.489112 3099 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: TxnStatusTablet state changed. Reason: New leader 17a4dc131fdf48df9947ba8ef3ae9b6a. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.489215 3099 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.489723 3097 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.489815 3097 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:07:42.490417 3099 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.490511 3099 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 315cbb9bb61e493fac27d2d13300b6ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.490614 3120 mvcc.cc:205] Tried to move back new op lower bound from 7333460633545166848 to 7333460633477988352. Current Snapshot: MvccSnapshot[applied={T|T < 7333460633545166848}]
I20260926 04:07:42.491882 3097 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.492044 3097 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P 17a4dc131fdf48df9947ba8ef3ae9b6a: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:07:42.492630 3098 tablet_replica.cc:439] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } } }
I20260926 04:07:42.492717 3098 tablet_replica.cc:442] T 68214f85d8ba499b8186bf10d53a2c38 P dce473b2bc8043e58b7f959e7ee35536: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:07:42.494908 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.492260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.213.194:35345)
0926 04:07:42.492338 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.492340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.492348 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:42.492373 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.492374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.492390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.492461 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:42.492654 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.493315 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.494141 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.494346 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.494420 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.494586 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.494694 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.494784 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.494808 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.494830 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 04:07:42.499897 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.497430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:52821 (local address 127.0.213.193:36869)
0926 04:07:42.497474 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.497476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.497485 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:42.497553 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.497554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.497565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.497619 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:42.497785 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.498254 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.499161 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.499345 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.499429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.499632 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.499665 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.499735 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.499800 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.499819 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:07:42.499903 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.497358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36869 (local address 127.0.213.194:52821)
0926 04:07:42.497431 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.497442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.497496 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.497649 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.497650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.497651 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:42.497710 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.497711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.498336 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.498337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.499033 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.499035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.499431 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.499432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.499497 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.499519 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.499740 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.499742 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.499795 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.499838 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 04:07:42.502496 3022 raft_consensus.cc:1261] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Refusing update from remote peer dce473b2bc8043e58b7f959e7ee35536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:42.502555 2742 raft_consensus.cc:1261] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Refusing update from remote peer dce473b2bc8043e58b7f959e7ee35536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:42.502813 3101 consensus_queue.cc:1035] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:42.502956 3098 consensus_queue.cc:1035] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:42.504375 3119 mvcc.cc:205] Tried to move back new op lower bound from 7333460633608822784 to 7333460633106350080. Current Snapshot: MvccSnapshot[applied={T|T < 7333460633608822784}]
I20260926 04:07:42.504940 3118 mvcc.cc:205] Tried to move back new op lower bound from 7333460633608822784 to 7333460633106350080. Current Snapshot: MvccSnapshot[applied={T|T < 7333460633608822784}]
I20260926 04:07:42.511682 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.508831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:49565 (local address 127.0.213.194:35345)
0926 04:07:42.509117 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.509119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.509126 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:07:42.509147 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.509148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.509158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.509209 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:42.509372 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.509963 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.510897 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.511186 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.511267 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.511453 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.511484 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.511537 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.511596 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.511615 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260926 04:07:42.511730 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.508790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:35345 (local address 127.0.213.193:49565)
0926 04:07:42.508937 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.508948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.508996 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.509214 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.509214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.509216 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:42.509292 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.509294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.510041 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.510042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.510737 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.510740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.511268 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.511269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.511343 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.511363 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.511539 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.511541 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.511591 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.511638 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260926 04:07:42.521142 2585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:42.521277 2585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:42.522046 2585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Sending DeleteTablet for 3 replicas of tablet a98e428bc4cb4901b8efd66a0cbc9728
I20260926 04:07:42.522406 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet a98e428bc4cb4901b8efd66a0cbc9728 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:42 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:41332
I20260926 04:07:42.522491 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet a98e428bc4cb4901b8efd66a0cbc9728 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:42 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260926 04:07:42.522830 2722 tablet_service.cc:1514] Processing DeleteTablet for tablet a98e428bc4cb4901b8efd66a0cbc9728 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:42 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40558
I20260926 04:07:42.523161 3138 tablet_replica.cc:330] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: stopping tablet replica
I20260926 04:07:42.523115 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:42.523263 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:42.523272 3138 raft_consensus.cc:2229] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:42.523334 3138 raft_consensus.cc:2258] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:42.524672 2722 tablet_service.cc:1467] Processing CreateTablet for tablet 940b4d96933547ac98feb26f8f0e11a0 (DEFAULT_TABLE table=test-table [id=3b9fe6d170794cb0b683c562496eb366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:42.524672 3002 tablet_service.cc:1467] Processing CreateTablet for tablet 940b4d96933547ac98feb26f8f0e11a0 (DEFAULT_TABLE table=test-table [id=3b9fe6d170794cb0b683c562496eb366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:42.524801 2722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940b4d96933547ac98feb26f8f0e11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.524802 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940b4d96933547ac98feb26f8f0e11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.526129 3091 tablet_bootstrap.cc:492] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Bootstrap starting.
I20260926 04:07:42.526142 3092 tablet_bootstrap.cc:492] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Bootstrap starting.
I20260926 04:07:42.526528 3092 tablet_bootstrap.cc:654] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.526603 3091 tablet_bootstrap.cc:654] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.527168 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 940b4d96933547ac98feb26f8f0e11a0 (DEFAULT_TABLE table=test-table [id=3b9fe6d170794cb0b683c562496eb366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:42.527281 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940b4d96933547ac98feb26f8f0e11a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:42.527338 3092 tablet_bootstrap.cc:492] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: No bootstrap required, opened a new log
I20260926 04:07:42.527376 3092 ts_tablet_manager.cc:1397] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:42.527508 3092 raft_consensus.cc:348] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.527570 3092 raft_consensus.cc:374] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.527588 3092 raft_consensus.cc:729] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315cbb9bb61e493fac27d2d13300b6ea, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.527632 3092 consensus_queue.cc:260] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.527707 3092 ts_tablet_manager.cc:1428] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:42.528581 3138 ts_tablet_manager.cc:1905] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:42.529091 3094 tablet_bootstrap.cc:492] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Bootstrap starting.
I20260926 04:07:42.529475 3094 tablet_bootstrap.cc:654] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:42.530186 3094 tablet_bootstrap.cc:492] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: No bootstrap required, opened a new log
I20260926 04:07:42.530236 3094 ts_tablet_manager.cc:1397] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:42.530388 3094 raft_consensus.cc:348] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.530457 3094 raft_consensus.cc:374] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.530478 3094 raft_consensus.cc:729] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce473b2bc8043e58b7f959e7ee35536, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.530526 3094 consensus_queue.cc:260] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.530639 3094 ts_tablet_manager.cc:1428] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:42.531096 3139 tablet_replica.cc:330] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: stopping tablet replica
I20260926 04:07:42.531226 3139 raft_consensus.cc:2229] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:42.531315 3139 raft_consensus.cc:2258] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:42.530915 3137 tablet_replica.cc:330] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: stopping tablet replica
I20260926 04:07:42.532543 3139 ts_tablet_manager.cc:1905] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:42.533066 3137 raft_consensus.cc:2229] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:42.533228 3137 raft_consensus.cc:2258] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:42.533407 3138 ts_tablet_manager.cc:1918] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:42.533490 3138 log.cc:1196] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/a98e428bc4cb4901b8efd66a0cbc9728
I20260926 04:07:42.533751 3091 tablet_bootstrap.cc:492] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: No bootstrap required, opened a new log
I20260926 04:07:42.533766 3138 ts_tablet_manager.cc:1939] T a98e428bc4cb4901b8efd66a0cbc9728 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting consensus metadata
I20260926 04:07:42.533812 3091 ts_tablet_manager.cc:1397] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260926 04:07:42.533977 3091 raft_consensus.cc:348] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.534049 3091 raft_consensus.cc:374] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:42.534070 3091 raft_consensus.cc:729] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Initialized, Role: FOLLOWER
I20260926 04:07:42.534116 3091 consensus_queue.cc:260] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.534201 3091 ts_tablet_manager.cc:1428] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:42.534358 3137 ts_tablet_manager.cc:1905] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:42.535626 3139 ts_tablet_manager.cc:1918] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:42.535789 3139 log.cc:1196] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/a98e428bc4cb4901b8efd66a0cbc9728
I20260926 04:07:42.535974 2569 catalog_manager.cc:4841] TS 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669): tablet a98e428bc4cb4901b8efd66a0cbc9728 (table test-table [id=4c12bbefdac7444880d8a4a10d47467f]) successfully deleted
I20260926 04:07:42.536187 3139 ts_tablet_manager.cc:1939] T a98e428bc4cb4901b8efd66a0cbc9728 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting consensus metadata
I20260926 04:07:42.536618 2572 catalog_manager.cc:4841] TS 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869): tablet a98e428bc4cb4901b8efd66a0cbc9728 (table test-table [id=4c12bbefdac7444880d8a4a10d47467f]) successfully deleted
I20260926 04:07:42.536844 3137 ts_tablet_manager.cc:1918] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:42.536926 3137 log.cc:1196] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/a98e428bc4cb4901b8efd66a0cbc9728
I20260926 04:07:42.537159 3137 ts_tablet_manager.cc:1939] T a98e428bc4cb4901b8efd66a0cbc9728 P dce473b2bc8043e58b7f959e7ee35536: Deleting consensus metadata
I20260926 04:07:42.537571 2570 catalog_manager.cc:4841] TS dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345): tablet a98e428bc4cb4901b8efd66a0cbc9728 (table test-table [id=4c12bbefdac7444880d8a4a10d47467f]) successfully deleted
W20260926 04:07:42.556574 3069 tablet.cc:2369] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:07:42.861204 3132 raft_consensus.cc:482] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:42.861322 3132 raft_consensus.cc:504] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.861547 3132 leader_election.cc:288] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:42.861816 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940b4d96933547ac98feb26f8f0e11a0" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea" is_pre_election: true
I20260926 04:07:42.861816 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940b4d96933547ac98feb26f8f0e11a0" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536" is_pre_election: true
I20260926 04:07:42.861922 3022 raft_consensus.cc:2454] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:42.861922 2882 raft_consensus.cc:2454] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:42.862092 2674 leader_election.cc:302] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:42.862229 3132 raft_consensus.cc:2781] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:42.862277 3132 raft_consensus.cc:482] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:42.862291 3132 raft_consensus.cc:3037] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.863049 3132 raft_consensus.cc:504] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.863189 3132 leader_election.cc:288] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Requested vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:42.863349 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940b4d96933547ac98feb26f8f0e11a0" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536"
I20260926 04:07:42.863432 2882 raft_consensus.cc:3037] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.863371 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940b4d96933547ac98feb26f8f0e11a0" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea"
I20260926 04:07:42.863467 3022 raft_consensus.cc:3037] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:42.864115 2882 raft_consensus.cc:2454] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:42.864123 3022 raft_consensus.cc:2454] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:42.864307 2674 leader_election.cc:302] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, 315cbb9bb61e493fac27d2d13300b6ea, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:42.864413 3132 raft_consensus.cc:2781] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:42.864481 3132 raft_consensus.cc:686] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Becoming Leader. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Running, Role: LEADER
I20260926 04:07:42.864537 3132 consensus_queue.cc:237] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:42.865123 2584 catalog_manager.cc:5495] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193). New cstate: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:42.886255 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.883977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.0.213.193:36869)
0926 04:07:42.884023 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.884026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.884090 (+ 64us) server_negotiation.cc:407] Connection header received
0926 04:07:42.884182 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.884183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.884196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.884250 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:42.884349 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.884819 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.885572 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.885774 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.885849 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.885986 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.886024 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.886085 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.886146 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.886165 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:07:42.890635 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.888383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36869 (local address 127.0.213.193:59439)
0926 04:07:42.888464 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:42.888476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:42.888528 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:42.888656 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:42.888657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:42.888658 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:07:42.888691 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.888692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.889213 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.889213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.889878 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:42.889881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.890255 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.890255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:42.890305 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.890310 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.890449 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.890451 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:42.890487 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 04:07:42.890525 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 04:07:42.890638 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:42.888470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:59439 (local address 127.0.213.193:36869)
0926 04:07:42.888514 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:42.888515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:42.888522 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:07:42.888590 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:42.888591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:42.888597 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:42.888639 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:42.888736 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.889146 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.889970 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:42.890147 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:42.890214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:42.890374 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:42.890396 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:42.890432 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:42.890529 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 04:07:42.890547 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:07:42.892642 3022 raft_consensus.cc:1261] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:42.892689 2882 raft_consensus.cc:1261] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:42.892867 3132 consensus_queue.cc:1035] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:42.892982 3113 consensus_queue.cc:1035] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 04:07:42.897233 3118 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 04:07:44.416021 2585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:44.416154 2585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:44.417078 2585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Sending DeleteTablet for 3 replicas of tablet 940b4d96933547ac98feb26f8f0e11a0
I20260926 04:07:44.417382 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet 940b4d96933547ac98feb26f8f0e11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:44 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260926 04:07:44.417376 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 940b4d96933547ac98feb26f8f0e11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:44 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:41332
I20260926 04:07:44.417675 3167 tablet_replica.cc:330] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: stopping tablet replica
I20260926 04:07:44.417739 3167 raft_consensus.cc:2229] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:44.417783 3167 raft_consensus.cc:2258] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:44.418720 3167 ts_tablet_manager.cc:1905] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:44.419013 2721 tablet_service.cc:1514] Processing DeleteTablet for tablet 940b4d96933547ac98feb26f8f0e11a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:44 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40558
I20260926 04:07:44.419227 3168 tablet_replica.cc:330] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: stopping tablet replica
I20260926 04:07:44.419296 3168 raft_consensus.cc:2229] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:44.419339 3168 raft_consensus.cc:2258] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:44.419435 3170 tablet_replica.cc:330] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: stopping tablet replica
I20260926 04:07:44.419484 3170 raft_consensus.cc:2229] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:44.419584 3170 raft_consensus.cc:2258] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:44.419816 3170 ts_tablet_manager.cc:1905] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:44.420697 3167 ts_tablet_manager.cc:1918] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 04:07:44.420768 3167 log.cc:1196] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/940b4d96933547ac98feb26f8f0e11a0
I20260926 04:07:44.421015 3167 ts_tablet_manager.cc:1939] T 940b4d96933547ac98feb26f8f0e11a0 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting consensus metadata
I20260926 04:07:44.421347 3170 ts_tablet_manager.cc:1918] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 04:07:44.421419 3170 log.cc:1196] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/940b4d96933547ac98feb26f8f0e11a0
I20260926 04:07:44.421514 2569 catalog_manager.cc:4841] TS 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669): tablet 940b4d96933547ac98feb26f8f0e11a0 (table test-table [id=3b9fe6d170794cb0b683c562496eb366]) successfully deleted
I20260926 04:07:44.421677 3170 ts_tablet_manager.cc:1939] T 940b4d96933547ac98feb26f8f0e11a0 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting consensus metadata
I20260926 04:07:44.421988 2572 catalog_manager.cc:4841] TS 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869): tablet 940b4d96933547ac98feb26f8f0e11a0 (table test-table [id=3b9fe6d170794cb0b683c562496eb366]) successfully deleted
I20260926 04:07:44.423825 3168 ts_tablet_manager.cc:1905] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:44.425705 3168 ts_tablet_manager.cc:1918] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 04:07:44.425760 3168 log.cc:1196] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/940b4d96933547ac98feb26f8f0e11a0
I20260926 04:07:44.425923 3168 ts_tablet_manager.cc:1939] T 940b4d96933547ac98feb26f8f0e11a0 P dce473b2bc8043e58b7f959e7ee35536: Deleting consensus metadata
I20260926 04:07:44.426357 2570 catalog_manager.cc:4841] TS dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345): tablet 940b4d96933547ac98feb26f8f0e11a0 (table test-table [id=3b9fe6d170794cb0b683c562496eb366]) successfully deleted
Sep 26 04:07:44 dist-test-slave-xf24 krb5kdc[2540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 04:07:44 dist-test-slave-xf24 krb5kdc[2540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:44.446664 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:44.435780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.0.213.254:42851)
0926 04:07:44.435928 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:44.435930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:44.435945 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:07:44.436018 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:44.436019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:44.436029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:44.436162 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:44.436705 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:44.437212 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:44.437999 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:44.438203 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:44.438262 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:44.443868 (+ 5606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:44.443870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:44.443870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:44.443885 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:44.445470 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:44.445853 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:44.445854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:44.445855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:44.445920 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:44.446122 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:44.446123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:44.446123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:44.446244 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:44.446354 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:44.446458 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 04:07:44.446545 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260926 04:07:44.448421 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:44.448596 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:44.450045 3002 tablet_service.cc:1467] Processing CreateTablet for tablet d66064ac7892490791dc2e317935b4ff (DEFAULT_TABLE table=test-table [id=fd95ad8910674d4aa661ecedfe51b04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:44.450063 2721 tablet_service.cc:1467] Processing CreateTablet for tablet d66064ac7892490791dc2e317935b4ff (DEFAULT_TABLE table=test-table [id=fd95ad8910674d4aa661ecedfe51b04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:44.450063 2862 tablet_service.cc:1467] Processing CreateTablet for tablet d66064ac7892490791dc2e317935b4ff (DEFAULT_TABLE table=test-table [id=fd95ad8910674d4aa661ecedfe51b04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:44.450201 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66064ac7892490791dc2e317935b4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:44.450203 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66064ac7892490791dc2e317935b4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:44.450202 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66064ac7892490791dc2e317935b4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:44.451344 3178 tablet_bootstrap.cc:492] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Bootstrap starting.
I20260926 04:07:44.451689 3179 tablet_bootstrap.cc:492] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Bootstrap starting.
I20260926 04:07:44.451845 3178 tablet_bootstrap.cc:654] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:44.451965 3180 tablet_bootstrap.cc:492] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Bootstrap starting.
I20260926 04:07:44.452097 3179 tablet_bootstrap.cc:654] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:44.452345 3180 tablet_bootstrap.cc:654] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:44.452569 3178 tablet_bootstrap.cc:492] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: No bootstrap required, opened a new log
I20260926 04:07:44.452574 3179 tablet_bootstrap.cc:492] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: No bootstrap required, opened a new log
I20260926 04:07:44.452612 3178 ts_tablet_manager.cc:1397] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:44.452641 3179 ts_tablet_manager.cc:1397] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:44.452741 3178 raft_consensus.cc:348] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.452751 3179 raft_consensus.cc:348] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.452821 3179 raft_consensus.cc:374] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:44.452817 3178 raft_consensus.cc:374] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:44.452844 3178 raft_consensus.cc:729] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315cbb9bb61e493fac27d2d13300b6ea, State: Initialized, Role: FOLLOWER
I20260926 04:07:44.452841 3179 raft_consensus.cc:729] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce473b2bc8043e58b7f959e7ee35536, State: Initialized, Role: FOLLOWER
I20260926 04:07:44.452912 3179 consensus_queue.cc:260] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.452916 3178 consensus_queue.cc:260] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.452945 3180 tablet_bootstrap.cc:492] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: No bootstrap required, opened a new log
I20260926 04:07:44.452984 3180 ts_tablet_manager.cc:1397] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:44.453095 3178 ts_tablet_manager.cc:1428] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 04:07:44.453110 3179 ts_tablet_manager.cc:1428] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 04:07:44.453109 3180 raft_consensus.cc:348] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.453172 3180 raft_consensus.cc:374] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:44.453191 3180 raft_consensus.cc:729] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Initialized, Role: FOLLOWER
I20260926 04:07:44.453236 3180 consensus_queue.cc:260] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.453336 3180 ts_tablet_manager.cc:1428] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:44.555836 3150 raft_consensus.cc:482] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:44.555938 3150 raft_consensus.cc:504] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.556160 3150 leader_election.cc:288] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:44.556411 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66064ac7892490791dc2e317935b4ff" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536" is_pre_election: true
I20260926 04:07:44.556463 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66064ac7892490791dc2e317935b4ff" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea" is_pre_election: true
I20260926 04:07:44.556543 2882 raft_consensus.cc:2454] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:44.556555 3022 raft_consensus.cc:2454] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:44.556746 2674 leader_election.cc:302] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:44.556859 3150 raft_consensus.cc:2781] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:44.556891 3150 raft_consensus.cc:482] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:44.556917 3150 raft_consensus.cc:3037] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:44.557654 3150 raft_consensus.cc:504] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.557811 3150 leader_election.cc:288] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Requested vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:44.557945 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66064ac7892490791dc2e317935b4ff" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea"
I20260926 04:07:44.557956 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66064ac7892490791dc2e317935b4ff" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536"
I20260926 04:07:44.558040 3022 raft_consensus.cc:3037] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:44.558033 2882 raft_consensus.cc:3037] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:44.558679 3022 raft_consensus.cc:2454] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:44.558689 2882 raft_consensus.cc:2454] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:44.558899 2673 leader_election.cc:302] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, 315cbb9bb61e493fac27d2d13300b6ea; no voters:
I20260926 04:07:44.559006 3150 raft_consensus.cc:2781] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:44.559080 3150 raft_consensus.cc:686] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Becoming Leader. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Running, Role: LEADER
I20260926 04:07:44.559137 3150 consensus_queue.cc:237] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.559777 2585 catalog_manager.cc:5495] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193). New cstate: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:44.574182 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:44.571464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.0.213.193:36869)
0926 04:07:44.571620 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:44.571622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:44.571631 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:44.571659 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:44.571660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:44.571672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:44.571734 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:44.571937 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:44.572555 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:44.573413 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:44.573651 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:44.573731 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:44.573950 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:44.573985 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:44.574041 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:44.574101 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 04:07:44.574119 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":111,"threads_started":1}
I20260926 04:07:44.576557 3022 raft_consensus.cc:1261] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:44.576625 2882 raft_consensus.cc:1261] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:44.576787 3150 consensus_queue.cc:1035] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:44.577059 3150 consensus_queue.cc:1035] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:44.586772 2585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:44.586899 2585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:44.587707 2585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Sending DeleteTablet for 3 replicas of tablet d66064ac7892490791dc2e317935b4ff
I20260926 04:07:44.587975 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet d66064ac7892490791dc2e317935b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:44 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:41332
I20260926 04:07:44.588085 2722 tablet_service.cc:1514] Processing DeleteTablet for tablet d66064ac7892490791dc2e317935b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:44 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40558
I20260926 04:07:44.588208 3168 tablet_replica.cc:330] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: stopping tablet replica
I20260926 04:07:44.588207 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet d66064ac7892490791dc2e317935b4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:44 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260926 04:07:44.588238 3170 tablet_replica.cc:330] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: stopping tablet replica
I20260926 04:07:44.588295 3170 raft_consensus.cc:2229] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:44.588304 3168 raft_consensus.cc:2229] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:44.588349 3168 raft_consensus.cc:2258] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:44.588361 3170 raft_consensus.cc:2258] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:44.588428 3167 tablet_replica.cc:330] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: stopping tablet replica
I20260926 04:07:44.588500 3167 raft_consensus.cc:2229] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:44.588543 3167 raft_consensus.cc:2258] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:44.588658 3170 ts_tablet_manager.cc:1905] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:44.588997 3167 ts_tablet_manager.cc:1905] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:44.589054 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:44.589232 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:44.590656 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 (DEFAULT_TABLE table=test-table [id=be984f41ca54478ca31b6072cfdf90af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:44.590747 3002 tablet_service.cc:1467] Processing CreateTablet for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 (DEFAULT_TABLE table=test-table [id=be984f41ca54478ca31b6072cfdf90af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:44.590781 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:44.590809 3170 ts_tablet_manager.cc:1918] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:44.590840 3168 ts_tablet_manager.cc:1905] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:44.590865 3170 log.cc:1196] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/d66064ac7892490791dc2e317935b4ff
I20260926 04:07:44.591090 3170 ts_tablet_manager.cc:1939] T d66064ac7892490791dc2e317935b4ff P 17a4dc131fdf48df9947ba8ef3ae9b6a: Deleting consensus metadata
I20260926 04:07:44.591559 2572 catalog_manager.cc:4841] TS 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869): tablet d66064ac7892490791dc2e317935b4ff (table test-table [id=fd95ad8910674d4aa661ecedfe51b04d]) successfully deleted
I20260926 04:07:44.591813 3179 tablet_bootstrap.cc:492] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Bootstrap starting.
I20260926 04:07:44.592197 3179 tablet_bootstrap.cc:654] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:44.592340 3168 ts_tablet_manager.cc:1918] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:44.592399 3168 log.cc:1196] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/d66064ac7892490791dc2e317935b4ff
I20260926 04:07:44.592613 3168 ts_tablet_manager.cc:1939] T d66064ac7892490791dc2e317935b4ff P dce473b2bc8043e58b7f959e7ee35536: Deleting consensus metadata
I20260926 04:07:44.592819 3179 tablet_bootstrap.cc:492] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: No bootstrap required, opened a new log
I20260926 04:07:44.592798 2722 tablet_service.cc:1467] Processing CreateTablet for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 (DEFAULT_TABLE table=test-table [id=be984f41ca54478ca31b6072cfdf90af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:44.592885 3179 ts_tablet_manager.cc:1397] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:44.592932 2722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:44.592972 2570 catalog_manager.cc:4841] TS dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345): tablet d66064ac7892490791dc2e317935b4ff (table test-table [id=fd95ad8910674d4aa661ecedfe51b04d]) successfully deleted
I20260926 04:07:44.593045 3179 raft_consensus.cc:348] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.593140 3179 raft_consensus.cc:374] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:44.593169 3179 raft_consensus.cc:729] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce473b2bc8043e58b7f959e7ee35536, State: Initialized, Role: FOLLOWER
I20260926 04:07:44.593217 3179 consensus_queue.cc:260] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.593313 3179 ts_tablet_manager.cc:1428] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:44.594039 3180 tablet_bootstrap.cc:492] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Bootstrap starting.
I20260926 04:07:44.594304 3167 ts_tablet_manager.cc:1918] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:07:44.594353 3180 tablet_bootstrap.cc:654] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:44.594375 3167 log.cc:1196] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/d66064ac7892490791dc2e317935b4ff
I20260926 04:07:44.594623 3167 ts_tablet_manager.cc:1939] T d66064ac7892490791dc2e317935b4ff P 315cbb9bb61e493fac27d2d13300b6ea: Deleting consensus metadata
I20260926 04:07:44.594818 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:44.595003 3180 tablet_bootstrap.cc:492] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a: No bootstrap required, opened a new log
I20260926 04:07:44.595050 3180 ts_tablet_manager.cc:1397] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:44.595042 2569 catalog_manager.cc:4841] TS 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669): tablet d66064ac7892490791dc2e317935b4ff (table test-table [id=fd95ad8910674d4aa661ecedfe51b04d]) successfully deleted
I20260926 04:07:44.595176 3180 raft_consensus.cc:348] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.595245 3180 raft_consensus.cc:374] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:44.595264 3180 raft_consensus.cc:729] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Initialized, Role: FOLLOWER
I20260926 04:07:44.595305 3180 consensus_queue.cc:260] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.595395 3180 ts_tablet_manager.cc:1428] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:44.595930 3178 tablet_bootstrap.cc:492] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Bootstrap starting.
I20260926 04:07:44.596319 3178 tablet_bootstrap.cc:654] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:44.596968 3178 tablet_bootstrap.cc:492] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: No bootstrap required, opened a new log
I20260926 04:07:44.597034 3178 ts_tablet_manager.cc:1397] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:44.597182 3178 raft_consensus.cc:348] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.597259 3178 raft_consensus.cc:374] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:44.597283 3178 raft_consensus.cc:729] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315cbb9bb61e493fac27d2d13300b6ea, State: Initialized, Role: FOLLOWER
I20260926 04:07:44.597337 3178 consensus_queue.cc:260] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.597421 3178 ts_tablet_manager.cc:1428] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 04:07:44.744660 3155 raft_consensus.cc:482] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:44.744773 3155 raft_consensus.cc:504] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.744933 3155 leader_election.cc:288] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:44.745191 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add8e1a4f2d4d37bd2ce6fa4d1653b5" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea" is_pre_election: true
I20260926 04:07:44.745190 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add8e1a4f2d4d37bd2ce6fa4d1653b5" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536" is_pre_election: true
I20260926 04:07:44.745298 2882 raft_consensus.cc:2454] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:44.745298 3021 raft_consensus.cc:2454] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 0.
I20260926 04:07:44.745479 2673 leader_election.cc:302] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, 315cbb9bb61e493fac27d2d13300b6ea, dce473b2bc8043e58b7f959e7ee35536; no voters:
I20260926 04:07:44.745603 3155 raft_consensus.cc:2781] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:44.745653 3155 raft_consensus.cc:482] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:44.745687 3155 raft_consensus.cc:3037] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:44.746490 3155 raft_consensus.cc:504] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.746646 3155 leader_election.cc:288] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Requested vote from peers dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345), 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669)
I20260926 04:07:44.746793 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add8e1a4f2d4d37bd2ce6fa4d1653b5" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315cbb9bb61e493fac27d2d13300b6ea"
I20260926 04:07:44.746878 3021 raft_consensus.cc:3037] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:44.746845 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add8e1a4f2d4d37bd2ce6fa4d1653b5" candidate_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce473b2bc8043e58b7f959e7ee35536"
I20260926 04:07:44.746929 2882 raft_consensus.cc:3037] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:44.747646 3021 raft_consensus.cc:2454] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:44.747646 2882 raft_consensus.cc:2454] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4dc131fdf48df9947ba8ef3ae9b6a in term 1.
I20260926 04:07:44.747834 2673 leader_election.cc:302] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4dc131fdf48df9947ba8ef3ae9b6a, 315cbb9bb61e493fac27d2d13300b6ea; no voters:
I20260926 04:07:44.747938 3155 raft_consensus.cc:2781] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:44.748013 3155 raft_consensus.cc:686] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Becoming Leader. State: Replica: 17a4dc131fdf48df9947ba8ef3ae9b6a, State: Running, Role: LEADER
I20260926 04:07:44.748090 3155 consensus_queue.cc:237] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } }
I20260926 04:07:44.748785 2585 catalog_manager.cc:5495] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193). New cstate: current_term: 1 leader_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4dc131fdf48df9947ba8ef3ae9b6a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:44.783164 2882 raft_consensus.cc:1261] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:44.783176 3021 raft_consensus.cc:1261] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Refusing update from remote peer 17a4dc131fdf48df9947ba8ef3ae9b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:44.783397 3150 consensus_queue.cc:1035] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce473b2bc8043e58b7f959e7ee35536" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:44.783541 3155 consensus_queue.cc:1035] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "315cbb9bb61e493fac27d2d13300b6ea" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 04:07:44.788157 3148 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 04:07:46.305902 2585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:46.306010 2585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:46.306881 2585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebcb63327c7b466bbaeec3a176df65e2: Sending DeleteTablet for 3 replicas of tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5
I20260926 04:07:46.307150 2722 tablet_service.cc:1514] Processing DeleteTablet for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:46 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40558
I20260926 04:07:46.307250 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:46 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:41332
I20260926 04:07:46.307260 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:46 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42014
I20260926 04:07:46.307370 3218 tablet_replica.cc:330] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a: stopping tablet replica
I20260926 04:07:46.307435 3218 raft_consensus.cc:2229] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:46.307453 3219 tablet_replica.cc:330] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: stopping tablet replica
I20260926 04:07:46.307521 3218 raft_consensus.cc:2258] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 17a4dc131fdf48df9947ba8ef3ae9b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:46.307547 3219 raft_consensus.cc:2229] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:46.307552 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2657
I20260926 04:07:46.308194 3219 raft_consensus.cc:2258] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:46.308492 3219 ts_tablet_manager.cc:1905] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:46.310501 3219 ts_tablet_manager.cc:1918] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 04:07:46.310575 3219 log.cc:1196] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/3add8e1a4f2d4d37bd2ce6fa4d1653b5
I20260926 04:07:46.310799 3219 ts_tablet_manager.cc:1939] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P 315cbb9bb61e493fac27d2d13300b6ea: Deleting consensus metadata
I20260926 04:07:46.311542 2569 catalog_manager.cc:4841] TS 315cbb9bb61e493fac27d2d13300b6ea (127.0.213.195:42669): tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 (table test-table [id=be984f41ca54478ca31b6072cfdf90af]) successfully deleted
I20260926 04:07:46.312669 3220 tablet_replica.cc:330] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: stopping tablet replica
I20260926 04:07:46.312750 3220 raft_consensus.cc:2229] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:46.312795 3220 raft_consensus.cc:2258] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:46.313058 3220 ts_tablet_manager.cc:1905] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:46.314949 3220 ts_tablet_manager.cc:1918] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 04:07:46.315018 3220 log.cc:1196] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/3add8e1a4f2d4d37bd2ce6fa4d1653b5
I20260926 04:07:46.315236 3220 ts_tablet_manager.cc:1939] T 3add8e1a4f2d4d37bd2ce6fa4d1653b5 P dce473b2bc8043e58b7f959e7ee35536: Deleting consensus metadata
W20260926 04:07:46.315848 2572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:36869 (error 108)
I20260926 04:07:46.316044 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2797
I20260926 04:07:46.318393 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:46.318059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36869 (local address 127.0.0.1:35524)
0926 04:07:46.318271 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:46.318334 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:36869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":127,"threads_started":1}
W20260926 04:07:46.318573 2572 catalog_manager.cc:4584] TS 17a4dc131fdf48df9947ba8ef3ae9b6a (127.0.213.193:36869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5: Network error: Client connection negotiation failed: client connection to 127.0.213.193:36869: connect: Connection refused (error 111)
I20260926 04:07:46.322417 2570 catalog_manager.cc:4841] TS dce473b2bc8043e58b7f959e7ee35536 (127.0.213.194:35345): tablet 3add8e1a4f2d4d37bd2ce6fa4d1653b5 (table test-table [id=be984f41ca54478ca31b6072cfdf90af]) successfully deleted
I20260926 04:07:46.322715 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2937
I20260926 04:07:46.328207 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 2553
2026-09-26T04:07:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5841 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:46 dist-test-slave-xf24 krb5kdc[3225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:46 dist-test-slave-xf24 krb5kdc[3225](info): set up 2 sockets
Sep 26 04:07:46 dist-test-slave-xf24 krb5kdc[3225](info): commencing operation
krb5kdc: starting...
W20260926 04:07:47.388023 855 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:47 dist-test-slave-xf24 krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:47.526413 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34731
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:34731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:47.607436 3238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:47.607704 3238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:47.607750 3238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:47.609023 3238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:47.609109 3238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:47.609133 3238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:47.609155 3238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:47.609189 3238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:47.610781 3238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:34731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:47.611080 3238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:47.611321 3238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:47.616564 3244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:47.616566 3243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:47.616818 3246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:47.617100 3238 server_base.cc:1034] running on GCE node
I20260926 04:07:47.617309 3238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:47.617568 3238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:47.618808 3238 hybrid_clock.cc:648] HybridClock initialized: now 1790395667618791 us; error 43 us; skew 500 ppm
Sep 26 04:07:47 dist-test-slave-xf24 krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:47.621788 3238 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:47.622241 3238 webserver.cc:460] Webserver started at http://127.0.213.254:35243/ using document root <none> and password file <none>
I20260926 04:07:47.622493 3238 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:47.622543 3238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:47.622687 3238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:47.623847 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6"
format_stamp: "Formatted at 2026-09-26 04:07:47 on dist-test-slave-xf24"
server_key: "9e1543e61b853e3deb90d402b9580310"
server_key_iv: "c8a4eac4917fdcabd445a035bfbaaf63"
server_key_version: "encryptionkey@0"
I20260926 04:07:47.624266 3238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6"
format_stamp: "Formatted at 2026-09-26 04:07:47 on dist-test-slave-xf24"
server_key: "9e1543e61b853e3deb90d402b9580310"
server_key_iv: "c8a4eac4917fdcabd445a035bfbaaf63"
server_key_version: "encryptionkey@0"
I20260926 04:07:47.625818 3238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:47.626711 3253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:47.627007 3238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:47.627096 3238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6"
format_stamp: "Formatted at 2026-09-26 04:07:47 on dist-test-slave-xf24"
server_key: "9e1543e61b853e3deb90d402b9580310"
server_key_iv: "c8a4eac4917fdcabd445a035bfbaaf63"
server_key_version: "encryptionkey@0"
I20260926 04:07:47.627199 3238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:47.644804 3238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:47.647303 3238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:47.647459 3238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:47.651897 3238 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:34731
I20260926 04:07:47.652293 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:34731 every 8 connection(s)
I20260926 04:07:47.652361 3238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:47.653208 3306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:47.655870 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6: Bootstrap starting.
I20260926 04:07:47.656595 3306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:47.656955 3306 log.cc:824] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:47.657848 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6: No bootstrap required, opened a new log
I20260926 04:07:47.659687 3306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34731 } }
I20260926 04:07:47.659866 3306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:47.659898 3306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c44d7ad49f4b9386ce00fd4c6f90d6, State: Initialized, Role: FOLLOWER
I20260926 04:07:47.660136 3306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34731 } }
I20260926 04:07:47.660216 3306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:47.660243 3306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:47.660274 3306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:47.661269 3306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34731 } }
I20260926 04:07:47.661396 3306 leader_election.cc:302] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c44d7ad49f4b9386ce00fd4c6f90d6; no voters:
I20260926 04:07:47.661612 3306 leader_election.cc:288] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:47.661667 3309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:47.661975 3306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:47.662048 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3238
I20260926 04:07:47.662148 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:47.662303 855 external_mini_cluster.cc:1428] Setting key b43f69cc31af1417c1bafe289372293a
I20260926 04:07:47.662592 3309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [term 1 LEADER]: Becoming Leader. State: Replica: 12c44d7ad49f4b9386ce00fd4c6f90d6, State: Running, Role: LEADER
I20260926 04:07:47.662755 3309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34731 } }
I20260926 04:07:47.663206 3311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c44d7ad49f4b9386ce00fd4c6f90d6. Latest consensus state: current_term: 1 leader_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34731 } } }
I20260926 04:07:47.663295 3311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:47.663517 3320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:47.663657 3310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c44d7ad49f4b9386ce00fd4c6f90d6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34731 } } }
I20260926 04:07:47.663740 3310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:47.664039 3320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:47.666435 3320 catalog_manager.cc:1347] Generated new cluster ID: ee50143fd2f2482f9a5fe13050d99881
I20260926 04:07:47.666499 3320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:07:47 dist-test-slave-xf24 krb5kdc[3225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:47.675817 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:47.662987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.0.213.254:34731)
0926 04:07:47.663957 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:47.663970 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:47.664011 (+ 41us) server_negotiation.cc:407] Connection header received
0926 04:07:47.664177 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:47.664185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:47.664272 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:47.664451 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:47.665185 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:47.666265 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:47.667366 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:47.668129 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:47.668268 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:47.671277 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:47.671282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:47.671283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:47.671298 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:47.673823 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:47.674308 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:47.674312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:47.674312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:47.674439 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:47.674729 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:47.674730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:47.674731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:47.675115 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:47.675292 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:47.675381 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 04:07:47.675545 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":78,"threads_started":1}
I20260926 04:07:47.695511 3320 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:47.696391 3320 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:47.721674 3320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c44d7ad49f4b9386ce00fd4c6f90d6: Generated new TSK 0
I20260926 04:07:47.721982 3320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:47.755218 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34731
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:47.852763 3332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:47.852999 3332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:47.853027 3332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:47.854223 3332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:47.854266 3332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:47.854293 3332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:47.855813 3332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:47.856020 3332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:47.856300 3332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:07:47.861706 3332 server_base.cc:1034] running on GCE node
W20260926 04:07:47.861675 3338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:47.861675 3337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:47.861675 3340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:47.862149 3332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:47.862429 3332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:47.863613 3332 hybrid_clock.cc:648] HybridClock initialized: now 1790395667863602 us; error 36 us; skew 500 ppm
Sep 26 04:07:47 dist-test-slave-xf24 krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:47.866319 3332 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:47.866778 3332 webserver.cc:460] Webserver started at http://127.0.213.193:40731/ using document root <none> and password file <none>
I20260926 04:07:47.867017 3332 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:47.867085 3332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:47.867255 3332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:47.868443 3332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "989c9c208d684768807a68e7fd5b800f"
format_stamp: "Formatted at 2026-09-26 04:07:47 on dist-test-slave-xf24"
server_key: "b141191690d7e680aa91912cc8527c0b"
server_key_iv: "3d70443dd2ccadc36da8dbf8b5e68a01"
server_key_version: "encryptionkey@0"
I20260926 04:07:47.868862 3332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "989c9c208d684768807a68e7fd5b800f"
format_stamp: "Formatted at 2026-09-26 04:07:47 on dist-test-slave-xf24"
server_key: "b141191690d7e680aa91912cc8527c0b"
server_key_iv: "3d70443dd2ccadc36da8dbf8b5e68a01"
server_key_version: "encryptionkey@0"
I20260926 04:07:47.870386 3332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:47.871411 3347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:47.871749 3332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:47.871897 3332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "989c9c208d684768807a68e7fd5b800f"
format_stamp: "Formatted at 2026-09-26 04:07:47 on dist-test-slave-xf24"
server_key: "b141191690d7e680aa91912cc8527c0b"
server_key_iv: "3d70443dd2ccadc36da8dbf8b5e68a01"
server_key_version: "encryptionkey@0"
I20260926 04:07:47.871991 3332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:47.921443 3332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:47.924089 3332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:47.924301 3332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:47.924572 3332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:47.924917 3332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:47.924978 3332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:47.925024 3332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:47.925064 3332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:47.931043 3332 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:36309
I20260926 04:07:47.931085 3460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:36309 every 8 connection(s)
I20260926 04:07:47.931471 3332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:47.931985 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3332
I20260926 04:07:47.932107 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:47.932329 855 external_mini_cluster.cc:1428] Setting key 9b6b333cbafdccaa80bbbb06e2785621
Sep 26 04:07:47 dist-test-slave-xf24 krb5kdc[3225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:07:47.942314 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:47.932421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:59529 (local address 127.0.213.254:34731)
0926 04:07:47.932490 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:47.932493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:47.932727 (+ 234us) server_negotiation.cc:407] Connection header received
0926 04:07:47.933756 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:47.933758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:47.933816 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:47.933866 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:47.934579 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:47.935086 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:47.936158 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:47.936381 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:47.936458 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:47.939176 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:47.939178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:47.939178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:47.939187 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:47.940744 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:47.941244 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:47.941246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:47.941246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:47.941317 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:47.941585 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:47.941586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:47.941586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:47.941719 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:47.941792 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:47.942101 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 04:07:47.942195 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:07:47.942592 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:47.932416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:34731 (local address 127.0.213.193:59529)
0926 04:07:47.932602 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:47.932644 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:47.933636 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:47.933965 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:47.933976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:47.934265 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:47.934469 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:47.934474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:47.935184 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:47.935185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:47.936014 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:47.936016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:47.936478 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:47.936479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:47.936629 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:47.937313 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:47.937324 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:47.939059 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:47.940842 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:47.940852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:47.940855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:47.941097 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:47.941393 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:47.941394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:47.941394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:47.941516 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:47.941802 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:47.941810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:47.941960 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 04:07:47.942079 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 04:07:47.942287 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
I20260926 04:07:47.943215 3461 heartbeater.cc:347] Connected to a master server at 127.0.213.254:34731
I20260926 04:07:47.943344 3461 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:47.943572 3461 heartbeater.cc:511] Master 127.0.213.254:34731 requested a full tablet report, sending...
I20260926 04:07:47.944209 3270 ts_manager.cc:194] Registered new tserver with Master: 989c9c208d684768807a68e7fd5b800f (127.0.213.193:36309)
I20260926 04:07:47.944829 3270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:59529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:47.959622 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34731
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:48.050997 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:48.051256 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:48.051285 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:48.052711 3466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:48.052767 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:48.052800 3466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:48.054566 3466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:48.054803 3466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:48.055008 3466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:48.060374 3471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:48.060375 3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:48.060374 3474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:48.060801 3466 server_base.cc:1034] running on GCE node
I20260926 04:07:48.061088 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:48.061432 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:48.062599 3466 hybrid_clock.cc:648] HybridClock initialized: now 1790395668062585 us; error 50 us; skew 500 ppm
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:48.065548 3466 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:48.066133 3466 webserver.cc:460] Webserver started at http://127.0.213.194:35067/ using document root <none> and password file <none>
I20260926 04:07:48.066387 3466 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:48.066464 3466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:48.066633 3466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:48.067823 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "a11883e488d849d19889ec96dce6a85d"
format_stamp: "Formatted at 2026-09-26 04:07:48 on dist-test-slave-xf24"
server_key: "067db0788612fab69b5e66665f0f0b61"
server_key_iv: "1855c24e9c744530895a74e2c9bc5b97"
server_key_version: "encryptionkey@0"
I20260926 04:07:48.068272 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "a11883e488d849d19889ec96dce6a85d"
format_stamp: "Formatted at 2026-09-26 04:07:48 on dist-test-slave-xf24"
server_key: "067db0788612fab69b5e66665f0f0b61"
server_key_iv: "1855c24e9c744530895a74e2c9bc5b97"
server_key_version: "encryptionkey@0"
I20260926 04:07:48.069862 3466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:48.070809 3481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:48.071013 3466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:48.071133 3466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "a11883e488d849d19889ec96dce6a85d"
format_stamp: "Formatted at 2026-09-26 04:07:48 on dist-test-slave-xf24"
server_key: "067db0788612fab69b5e66665f0f0b61"
server_key_iv: "1855c24e9c744530895a74e2c9bc5b97"
server_key_version: "encryptionkey@0"
I20260926 04:07:48.071234 3466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:48.079969 3466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:48.081952 3466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:48.082125 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:48.082369 3466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:48.082691 3466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:48.082747 3466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:48.082790 3466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:48.082815 3466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:48.088939 3466 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:34611
I20260926 04:07:48.088994 3594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:34611 every 8 connection(s)
I20260926 04:07:48.089342 3466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:48.095347 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3466
I20260926 04:07:48.095448 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:48.095651 855 external_mini_cluster.cc:1428] Setting key 2c579a52ac38d09cb1744c4c7525214b
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:48.099788 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:48.090145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:41755 (local address 127.0.213.254:34731)
0926 04:07:48.090210 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:48.090212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:48.090390 (+ 178us) server_negotiation.cc:407] Connection header received
0926 04:07:48.091335 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:48.091336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:48.091345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:48.091409 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:48.091986 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.092555 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.093558 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.093741 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.093796 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:48.096512 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:48.096513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:48.096514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:48.096527 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:48.098169 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:48.098702 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:48.098703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:48.098703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:48.098755 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:48.099045 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:48.099046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:48.099046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:48.099148 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:48.099207 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:48.099545 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 04:07:48.099645 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:07:48.099797 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:48.090108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:34731 (local address 127.0.213.194:41755)
0926 04:07:48.090262 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:48.090303 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:48.091238 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:48.091465 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:48.091468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:48.091718 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:48.091916 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:48.091920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.092646 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.092648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:48.093477 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:48.093479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.093876 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.093877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:48.094003 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:48.094560 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:48.094569 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:48.096400 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:48.098268 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:48.098276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:48.098278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:48.098609 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:48.098822 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:48.098823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:48.098823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:48.098973 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:48.099220 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:48.099225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:48.099395 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 04:07:48.099488 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 04:07:48.099623 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260926 04:07:48.100442 3595 heartbeater.cc:347] Connected to a master server at 127.0.213.254:34731
I20260926 04:07:48.100559 3595 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:48.100775 3595 heartbeater.cc:511] Master 127.0.213.254:34731 requested a full tablet report, sending...
I20260926 04:07:48.101186 3270 ts_manager.cc:194] Registered new tserver with Master: a11883e488d849d19889ec96dce6a85d (127.0.213.194:34611)
I20260926 04:07:48.101568 3270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:41755
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:48.123029 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34731
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:48.207355 3600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:48.207650 3600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:48.207679 3600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:48.208871 3600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:48.208911 3600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:48.208938 3600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:48.210407 3600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:34731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:48.210603 3600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:48.210791 3600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:48.215958 3608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:48.215929 3605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:48.215961 3600 server_base.cc:1034] running on GCE node
W20260926 04:07:48.215930 3606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:48.216500 3600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:48.216818 3600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:48.218010 3600 hybrid_clock.cc:648] HybridClock initialized: now 1790395668217995 us; error 31 us; skew 500 ppm
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:48.220754 3600 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:48.221220 3600 webserver.cc:460] Webserver started at http://127.0.213.195:42049/ using document root <none> and password file <none>
I20260926 04:07:48.221470 3600 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:48.221539 3600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:48.221724 3600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:48.222790 3600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "0576e579e05940d592f04495950b6315"
format_stamp: "Formatted at 2026-09-26 04:07:48 on dist-test-slave-xf24"
server_key: "79361917cec71c07edea3cef7f8a7d33"
server_key_iv: "27405b0334fc0c7799822cd6638ac258"
server_key_version: "encryptionkey@0"
I20260926 04:07:48.223152 3600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "0576e579e05940d592f04495950b6315"
format_stamp: "Formatted at 2026-09-26 04:07:48 on dist-test-slave-xf24"
server_key: "79361917cec71c07edea3cef7f8a7d33"
server_key_iv: "27405b0334fc0c7799822cd6638ac258"
server_key_version: "encryptionkey@0"
I20260926 04:07:48.224538 3600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:48.225304 3615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:48.225471 3600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:07:48.225564 3600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "0576e579e05940d592f04495950b6315"
format_stamp: "Formatted at 2026-09-26 04:07:48 on dist-test-slave-xf24"
server_key: "79361917cec71c07edea3cef7f8a7d33"
server_key_iv: "27405b0334fc0c7799822cd6638ac258"
server_key_version: "encryptionkey@0"
I20260926 04:07:48.225689 3600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:48.239008 3600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:48.241626 3600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:48.241816 3600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:48.242084 3600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:48.242403 3600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:48.242457 3600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:48.242491 3600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:48.242519 3600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:48.248450 3600 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:39747
I20260926 04:07:48.248539 3728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:39747 every 8 connection(s)
I20260926 04:07:48.248926 3600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:48.258461 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3600
I20260926 04:07:48.258540 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:48.258774 855 external_mini_cluster.cc:1428] Setting key 531c333de4ed362dc7c016c555a05719
I20260926 04:07:48.260459 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:48.249803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:49189 (local address 127.0.213.254:34731)
0926 04:07:48.249884 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:48.249886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:48.250034 (+ 148us) server_negotiation.cc:407] Connection header received
0926 04:07:48.251078 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:48.251079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:48.251095 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:48.251166 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:48.251929 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.252496 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.253721 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.253922 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.253994 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:48.256820 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:48.256823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:48.256823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:48.256833 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:48.258635 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:48.259270 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:48.259272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:48.259272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:48.259345 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:48.259705 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:48.259706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:48.259706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:48.259880 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:48.259947 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:48.260325 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 04:07:48.260396 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 04:07:48.260754 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:48.249748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:34731 (local address 127.0.213.195:49189)
0926 04:07:48.249908 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:48.249947 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:48.250939 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:48.251254 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:48.251258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:48.251543 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:48.251811 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:48.251816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.252626 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.252627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:48.253572 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:48.253575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:48.254037 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:48.254039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:48.254189 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:48.254871 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:48.254913 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:48.256656 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:48.258739 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:48.258750 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:48.258753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:48.259162 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:48.259427 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:48.259428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:48.259428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:48.259621 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:48.259954 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:48.259961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:48.260183 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 04:07:48.260287 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 04:07:48.260432 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 04:07:48.261304 3729 heartbeater.cc:347] Connected to a master server at 127.0.213.254:34731
I20260926 04:07:48.261439 3729 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:48.261649 3729 heartbeater.cc:511] Master 127.0.213.254:34731 requested a full tablet report, sending...
I20260926 04:07:48.262105 3270 ts_manager.cc:194] Registered new tserver with Master: 0576e579e05940d592f04495950b6315 (127.0.213.195:39747)
I20260926 04:07:48.262826 3270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:49189
I20260926 04:07:48.270792 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 04:07:48.283041 3314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:48.281908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.213.254:34731)
0926 04:07:48.281982 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:48.281985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:48.282107 (+ 122us) server_negotiation.cc:407] Connection header received
0926 04:07:48.282191 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:48.282192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:48.282203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:48.282250 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:48.282981 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51748: BlockingRecv error: recv got EOF from 127.0.0.1:51748 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 04:07:48.283960 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3332
I20260926 04:07:48.288375 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3466
I20260926 04:07:48.292956 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3600
I20260926 04:07:48.297612 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3238
2026-09-26T04:07:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1967 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3741](info): set up 2 sockets
Sep 26 04:07:48 dist-test-slave-xf24 krb5kdc[3741](info): commencing operation
krb5kdc: starting...
W20260926 04:07:49.343983 855 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:49 dist-test-slave-xf24 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:49.467257 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36809
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:49.556633 3754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:49.556859 3754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:49.556881 3754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:49.558004 3754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:49.558044 3754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:49.558058 3754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:49.558071 3754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:49.558084 3754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:49.559604 3754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:36809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:49.559836 3754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:49.560038 3754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:49.565465 3762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:49.565482 3759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:49.565465 3760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:49.565728 3754 server_base.cc:1034] running on GCE node
I20260926 04:07:49.566000 3754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:49.566345 3754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:49.567526 3754 hybrid_clock.cc:648] HybridClock initialized: now 1790395669567509 us; error 48 us; skew 500 ppm
Sep 26 04:07:49 dist-test-slave-xf24 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:49.570547 3754 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:49.571095 3754 webserver.cc:460] Webserver started at http://127.0.213.254:40671/ using document root <none> and password file <none>
I20260926 04:07:49.571321 3754 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:49.571367 3754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:49.571478 3754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:49.572625 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "c2eb458d71564a2bb8cb33af7cf573da"
format_stamp: "Formatted at 2026-09-26 04:07:49 on dist-test-slave-xf24"
server_key: "a98c3d39231240ceecc789b7ef5b9cb1"
server_key_iv: "70c5b4bde35b03788d4c999a36879dda"
server_key_version: "encryptionkey@0"
I20260926 04:07:49.572937 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "c2eb458d71564a2bb8cb33af7cf573da"
format_stamp: "Formatted at 2026-09-26 04:07:49 on dist-test-slave-xf24"
server_key: "a98c3d39231240ceecc789b7ef5b9cb1"
server_key_iv: "70c5b4bde35b03788d4c999a36879dda"
server_key_version: "encryptionkey@0"
I20260926 04:07:49.574291 3754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:49.575095 3769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:49.575325 3754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:49.575400 3754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "c2eb458d71564a2bb8cb33af7cf573da"
format_stamp: "Formatted at 2026-09-26 04:07:49 on dist-test-slave-xf24"
server_key: "a98c3d39231240ceecc789b7ef5b9cb1"
server_key_iv: "70c5b4bde35b03788d4c999a36879dda"
server_key_version: "encryptionkey@0"
I20260926 04:07:49.575510 3754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:49.600286 3754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:49.602798 3754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:49.603104 3754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:49.607779 3754 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:36809
I20260926 04:07:49.607851 3821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:36809 every 8 connection(s)
I20260926 04:07:49.608284 3754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:49.609338 3822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:49.611740 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da: Bootstrap starting.
I20260926 04:07:49.612449 3822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:49.612747 3822 log.cc:824] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:49.613281 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3754
I20260926 04:07:49.613390 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:49.613574 855 external_mini_cluster.cc:1428] Setting key 83a6171309386ae4c6eda39dc571b69b
I20260926 04:07:49.613619 3822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da: No bootstrap required, opened a new log
I20260926 04:07:49.616662 3822 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eb458d71564a2bb8cb33af7cf573da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36809 } }
I20260926 04:07:49.616820 3822 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:49.616844 3822 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2eb458d71564a2bb8cb33af7cf573da, State: Initialized, Role: FOLLOWER
I20260926 04:07:49.616983 3822 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eb458d71564a2bb8cb33af7cf573da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36809 } }
I20260926 04:07:49.617061 3822 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:49.617085 3822 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:49.617157 3822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:49.617743 3822 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eb458d71564a2bb8cb33af7cf573da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36809 } }
I20260926 04:07:49.617841 3822 leader_election.cc:302] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2eb458d71564a2bb8cb33af7cf573da; no voters:
I20260926 04:07:49.618077 3822 leader_election.cc:288] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:49.618146 3827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:49.618304 3827 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [term 1 LEADER]: Becoming Leader. State: Replica: c2eb458d71564a2bb8cb33af7cf573da, State: Running, Role: LEADER
I20260926 04:07:49.618374 3822 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:49.618450 3827 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eb458d71564a2bb8cb33af7cf573da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36809 } }
I20260926 04:07:49.618801 3829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2eb458d71564a2bb8cb33af7cf573da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eb458d71564a2bb8cb33af7cf573da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36809 } } }
I20260926 04:07:49.618929 3829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:49.618803 3827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2eb458d71564a2bb8cb33af7cf573da. Latest consensus state: current_term: 1 leader_uuid: "c2eb458d71564a2bb8cb33af7cf573da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eb458d71564a2bb8cb33af7cf573da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36809 } } }
I20260926 04:07:49.619159 3834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:49.619585 3827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:49.619843 3834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 04:07:49 dist-test-slave-xf24 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:49.621766 3834 catalog_manager.cc:1347] Generated new cluster ID: 3365c4832dcc49949361e8c5134a8ac2
I20260926 04:07:49.621817 3834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:49.626011 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:49.614094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.0.213.254:36809)
0926 04:07:49.614286 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:49.614294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:49.614320 (+ 26us) server_negotiation.cc:407] Connection header received
0926 04:07:49.614978 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:49.614985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:49.615043 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:49.615171 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:49.616014 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:49.617093 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:49.617937 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:49.618585 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:49.619021 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:49.621937 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:49.621941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:49.621941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:49.621953 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:49.624011 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:49.624482 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:49.624485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:49.624486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:49.624616 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:49.624860 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:49.624861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:49.624861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:49.625167 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:49.625333 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:49.625468 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 04:07:49.625609 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 04:07:49.627010 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:07:49.644233 3834 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:49.645054 3834 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:49.652263 3834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2eb458d71564a2bb8cb33af7cf573da: Generated new TSK 0
I20260926 04:07:49.652515 3834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:07:49.652825 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:49.648887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.0.213.254:36809)
0926 04:07:49.648935 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:49.648937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:49.649074 (+ 137us) server_negotiation.cc:407] Connection header received
0926 04:07:49.649160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:49.649161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:49.649172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:49.649220 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:49.649644 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:49.650225 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:49.651044 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:49.651281 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:49.651367 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:49.652224 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:49.652226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:49.652226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:49.652510 (+ 284us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:49.652531 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:49.652539 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:49.652605 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:49.652700 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 04:07:49.652734 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:07:49.655572 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3754
2026-09-26T04:07:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1355 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:49 dist-test-slave-xf24 krb5kdc[3855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:49 dist-test-slave-xf24 krb5kdc[3855](info): set up 2 sockets
Sep 26 04:07:49 dist-test-slave-xf24 krb5kdc[3855](info): commencing operation
krb5kdc: starting...
W20260926 04:07:50.725039 855 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:50 dist-test-slave-xf24 krb5kdc[3855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:50.869164 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35433
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:35433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:50.957324 3868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:50.957571 3868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:50.957597 3868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:50.958772 3868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:50.958818 3868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:50.958833 3868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:50.958846 3868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:50.958859 3868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:50.960399 3868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:35433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:50.960626 3868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:50.960809 3868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:50.966305 3876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:50.966337 3874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:50.966337 3873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:50.966486 3868 server_base.cc:1034] running on GCE node
I20260926 04:07:50.966915 3868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:50.967314 3868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:50.968473 3868 hybrid_clock.cc:648] HybridClock initialized: now 1790395670968446 us; error 42 us; skew 500 ppm
Sep 26 04:07:50 dist-test-slave-xf24 krb5kdc[3855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:50.971424 3868 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:50.971977 3868 webserver.cc:460] Webserver started at http://127.0.213.254:40799/ using document root <none> and password file <none>
I20260926 04:07:50.972254 3868 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:50.972306 3868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:50.972441 3868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:50.973548 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "9c556aba6075438a811bba9a15866a53"
format_stamp: "Formatted at 2026-09-26 04:07:50 on dist-test-slave-xf24"
server_key: "1938e13875621927f4889850faa56429"
server_key_iv: "4db8797712590f4f536b9245a982ada7"
server_key_version: "encryptionkey@0"
I20260926 04:07:50.973919 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "9c556aba6075438a811bba9a15866a53"
format_stamp: "Formatted at 2026-09-26 04:07:50 on dist-test-slave-xf24"
server_key: "1938e13875621927f4889850faa56429"
server_key_iv: "4db8797712590f4f536b9245a982ada7"
server_key_version: "encryptionkey@0"
I20260926 04:07:50.975366 3868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:50.976307 3883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:50.976560 3868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:50.976680 3868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "9c556aba6075438a811bba9a15866a53"
format_stamp: "Formatted at 2026-09-26 04:07:50 on dist-test-slave-xf24"
server_key: "1938e13875621927f4889850faa56429"
server_key_iv: "4db8797712590f4f536b9245a982ada7"
server_key_version: "encryptionkey@0"
I20260926 04:07:50.976761 3868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:50.987376 3868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:50.990011 3868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:50.990212 3868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:50.994627 3868 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:35433
I20260926 04:07:50.994686 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:35433 every 8 connection(s)
I20260926 04:07:50.995275 3868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:50.996423 3936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:50.998819 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53: Bootstrap starting.
I20260926 04:07:50.999450 3936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:50.999873 3936 log.cc:824] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:51.000823 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53: No bootstrap required, opened a new log
I20260926 04:07:51.002717 3936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c556aba6075438a811bba9a15866a53" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35433 } }
I20260926 04:07:51.002911 3936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:51.002936 3936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c556aba6075438a811bba9a15866a53, State: Initialized, Role: FOLLOWER
I20260926 04:07:51.003132 3936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c556aba6075438a811bba9a15866a53" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35433 } }
I20260926 04:07:51.003230 3936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:51.003257 3936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:51.003285 3936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:51.004340 3936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c556aba6075438a811bba9a15866a53" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35433 } }
I20260926 04:07:51.004454 3936 leader_election.cc:302] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c556aba6075438a811bba9a15866a53; no voters:
I20260926 04:07:51.004645 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3868
I20260926 04:07:51.004698 3936 leader_election.cc:288] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:51.004729 3939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:51.004761 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:51.004892 3939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [term 1 LEADER]: Becoming Leader. State: Replica: 9c556aba6075438a811bba9a15866a53, State: Running, Role: LEADER
I20260926 04:07:51.004966 855 external_mini_cluster.cc:1428] Setting key 3312cb125f48330ddea2b27ad08f4e03
I20260926 04:07:51.005002 3936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:51.005024 3939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c556aba6075438a811bba9a15866a53" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35433 } }
I20260926 04:07:51.005347 3940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c556aba6075438a811bba9a15866a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c556aba6075438a811bba9a15866a53" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35433 } } }
I20260926 04:07:51.005414 3940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:51.005537 3940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c556aba6075438a811bba9a15866a53. Latest consensus state: current_term: 1 leader_uuid: "9c556aba6075438a811bba9a15866a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c556aba6075438a811bba9a15866a53" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35433 } } }
I20260926 04:07:51.005610 3940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:51.006358 3945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:51.006946 3945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:51.008716 3945 catalog_manager.cc:1347] Generated new cluster ID: 9f3d3bd4c2fb4079846c4878b591a41b
I20260926 04:07:51.008777 3945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:51.019382 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.005715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.0.213.254:35433)
0926 04:07:51.005976 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:51.005988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:51.006262 (+ 274us) server_negotiation.cc:407] Connection header received
0926 04:07:51.006424 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:51.006433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:51.006503 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:51.006690 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:51.007520 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.009128 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.010247 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.010927 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.011064 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.014360 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:51.014368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:51.014368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:51.014385 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:51.017419 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.017940 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.017945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.017946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.018089 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.018315 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.018316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.018316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.018683 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:51.018860 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:51.018968 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 04:07:51.019123 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260926 04:07:51.023910 3945 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:51.024699 3945 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:51.032269 3945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c556aba6075438a811bba9a15866a53: Generated new TSK 0
I20260926 04:07:51.032558 3945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:51.061096 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35433
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:51.150942 3962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:51.151247 3962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:51.151294 3962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:51.152822 3962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:51.152927 3962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:51.152982 3962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:51.154646 3962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:51.154960 3962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:51.155227 3962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:51.160650 3968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:51.160650 3970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:51.160704 3967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:51.161119 3962 server_base.cc:1034] running on GCE node
I20260926 04:07:51.161432 3962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:51.161691 3962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:51.162884 3962 hybrid_clock.cc:648] HybridClock initialized: now 1790395671162880 us; error 42 us; skew 500 ppm
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:51.165961 3962 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:51.166517 3962 webserver.cc:460] Webserver started at http://127.0.213.193:37263/ using document root <none> and password file <none>
I20260926 04:07:51.166769 3962 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:51.166841 3962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:51.166998 3962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:51.168286 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "723a73ea0fe54f50b3a4a017ec677f67"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "082a549ad0432886af1ccb6767802d88"
server_key_iv: "60b2864f4180bf03a1ba6de4f459d5da"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.168668 3962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "723a73ea0fe54f50b3a4a017ec677f67"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "082a549ad0432886af1ccb6767802d88"
server_key_iv: "60b2864f4180bf03a1ba6de4f459d5da"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.170091 3962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:51.170940 3977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.171252 3962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:51.171367 3962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "723a73ea0fe54f50b3a4a017ec677f67"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "082a549ad0432886af1ccb6767802d88"
server_key_iv: "60b2864f4180bf03a1ba6de4f459d5da"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.171468 3962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:51.186364 3962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:51.188936 3962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:51.189121 3962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:51.189384 3962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:51.189747 3962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:51.189807 3962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.189851 3962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:51.189879 3962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.196103 3962 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:38253
I20260926 04:07:51.196143 4090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:38253 every 8 connection(s)
I20260926 04:07:51.196486 3962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:51.197031 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 3962
I20260926 04:07:51.197126 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:51.197295 855 external_mini_cluster.cc:1428] Setting key 22007eb0fa6902ac8536e14d4daa07a2
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
I20260926 04:07:51.208911 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.198053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:54765 (local address 127.0.213.254:35433)
0926 04:07:51.198132 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:51.198135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:51.198421 (+ 286us) server_negotiation.cc:407] Connection header received
0926 04:07:51.199221 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:51.199222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:51.199236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:51.199288 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:51.200531 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.201375 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.202473 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.202839 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.202936 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.205577 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:51.205578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:51.205579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:51.205594 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:51.207337 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.207804 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.207805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.207806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.207873 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.208173 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.208175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.208175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.208325 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:51.208393 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:51.208684 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 04:07:51.208809 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 04:07:51.209087 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.198038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35433 (local address 127.0.213.193:54765)
0926 04:07:51.198285 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:51.198338 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:51.199100 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:51.199370 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:51.199375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:51.200231 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:51.200433 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:51.200438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.201471 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.201471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:51.202382 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:51.202384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.202954 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.202956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:51.203086 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.203708 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:51.203717 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:51.205429 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:51.207445 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:51.207455 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:51.207458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:51.207717 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:51.207945 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:51.207946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:51.207946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:51.208051 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:51.208393 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:51.208398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:51.208537 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 04:07:51.208624 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:07:51.208838 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260926 04:07:51.209762 4091 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35433
I20260926 04:07:51.209905 4091 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:51.210171 4091 heartbeater.cc:511] Master 127.0.213.254:35433 requested a full tablet report, sending...
I20260926 04:07:51.210845 3900 ts_manager.cc:194] Registered new tserver with Master: 723a73ea0fe54f50b3a4a017ec677f67 (127.0.213.193:38253)
I20260926 04:07:51.211477 3900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:54765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:51.228197 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35433
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:51.311237 4096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:51.311475 4096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:51.311501 4096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:51.312804 4096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:51.312865 4096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:51.312896 4096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:51.314564 4096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:51.314888 4096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:51.315182 4096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:51.321142 4102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:51.321156 4101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:51.321365 4104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:51.321463 4096 server_base.cc:1034] running on GCE node
I20260926 04:07:51.321691 4096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:51.321965 4096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:51.323163 4096 hybrid_clock.cc:648] HybridClock initialized: now 1790395671323168 us; error 55 us; skew 500 ppm
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:51.326108 4096 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:51.326606 4096 webserver.cc:460] Webserver started at http://127.0.213.194:42507/ using document root <none> and password file <none>
I20260926 04:07:51.326862 4096 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:51.326934 4096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:51.327095 4096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:51.328321 4096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "4673ccafffc74f548eab2519652bbd27"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "7a539f2d15060c7306863c2f47342ee6"
server_key_iv: "ea23289ba20a65350908853974085d83"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.328724 4096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "4673ccafffc74f548eab2519652bbd27"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "7a539f2d15060c7306863c2f47342ee6"
server_key_iv: "ea23289ba20a65350908853974085d83"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.330206 4096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:51.331045 4111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.331311 4096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:51.331462 4096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "4673ccafffc74f548eab2519652bbd27"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "7a539f2d15060c7306863c2f47342ee6"
server_key_iv: "ea23289ba20a65350908853974085d83"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.331550 4096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:51.344028 4096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:51.346654 4096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:51.346872 4096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:51.347203 4096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:51.347572 4096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:51.347641 4096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.347682 4096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:51.347714 4096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.353679 4096 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38709
I20260926 04:07:51.353729 4224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38709 every 8 connection(s)
I20260926 04:07:51.354074 4096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:51.363502 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4096
I20260926 04:07:51.363605 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:51.363821 855 external_mini_cluster.cc:1428] Setting key 5079b5073f2c26592cac16056d1e04cc
I20260926 04:07:51.364190 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.354865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:33737 (local address 127.0.213.254:35433)
0926 04:07:51.354914 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:51.354916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:51.355138 (+ 222us) server_negotiation.cc:407] Connection header received
0926 04:07:51.355698 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:51.355700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:51.355712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:51.355770 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:51.356849 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.357357 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.358379 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.358573 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.358681 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.361041 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:51.361043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:51.361043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:51.361052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:51.362636 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.363070 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.363072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.363072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.363146 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.363416 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.363417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.363418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.363546 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:51.363625 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:51.363981 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 04:07:51.364096 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:07:51.364427 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.354822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35433 (local address 127.0.213.194:33737)
0926 04:07:51.354988 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:51.355037 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:51.355584 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:51.355860 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:51.355865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:51.356530 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:51.356735 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:51.356740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.357456 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.357457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:51.358270 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:51.358271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.358688 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.358689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:51.358823 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.359323 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:51.359332 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:51.360930 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:51.362734 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:51.362742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:51.362744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:51.362987 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:51.363218 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:51.363219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:51.363219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:51.363345 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:51.363656 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:51.363663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:51.363821 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 04:07:51.363945 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 04:07:51.364156 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260926 04:07:51.365181 4225 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35433
I20260926 04:07:51.365309 4225 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:51.365545 4225 heartbeater.cc:511] Master 127.0.213.254:35433 requested a full tablet report, sending...
I20260926 04:07:51.366032 3900 ts_manager.cc:194] Registered new tserver with Master: 4673ccafffc74f548eab2519652bbd27 (127.0.213.194:38709)
I20260926 04:07:51.366603 3900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:33737
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:51.392982 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35433
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:36613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:51.481242 4230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:51.481505 4230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:51.481549 4230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:51.482802 4230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:51.482877 4230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:51.482924 4230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:51.484579 4230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:36613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:51.484874 4230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:51.485132 4230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:51.490399 4238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:51.490401 4235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:51.490624 4236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:51.490506 4230 server_base.cc:1034] running on GCE node
I20260926 04:07:51.490896 4230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:51.491170 4230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:51.492364 4230 hybrid_clock.cc:648] HybridClock initialized: now 1790395671492316 us; error 67 us; skew 500 ppm
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:51.495239 4230 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:51.495756 4230 webserver.cc:460] Webserver started at http://127.0.213.195:41881/ using document root <none> and password file <none>
I20260926 04:07:51.495961 4230 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:51.496004 4230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:51.496161 4230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:51.497236 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "d26521e44bf74fb49314babac284db14"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "5c75042fdfd9d2d07b2649ca436c25aa"
server_key_iv: "0317afcc5b8d7b8365e696321e26308b"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.497557 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "d26521e44bf74fb49314babac284db14"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "5c75042fdfd9d2d07b2649ca436c25aa"
server_key_iv: "0317afcc5b8d7b8365e696321e26308b"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.499173 4230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:51.500000 4245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.500231 4230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:51.500331 4230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "d26521e44bf74fb49314babac284db14"
format_stamp: "Formatted at 2026-09-26 04:07:51 on dist-test-slave-xf24"
server_key: "5c75042fdfd9d2d07b2649ca436c25aa"
server_key_iv: "0317afcc5b8d7b8365e696321e26308b"
server_key_version: "encryptionkey@0"
I20260926 04:07:51.500435 4230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:51.522383 4230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:51.524950 4230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:51.525144 4230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:51.525403 4230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:51.525774 4230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:51.525833 4230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.525879 4230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:51.525913 4230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:51.532015 4230 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:42387
I20260926 04:07:51.532140 4358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:42387 every 8 connection(s)
I20260926 04:07:51.532480 4230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:51.538789 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4230
I20260926 04:07:51.538872 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:51.539062 855 external_mini_cluster.cc:1428] Setting key 765f2e05f5f3f8fa510c63e069460f80
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:51.543033 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.533328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:47853 (local address 127.0.213.254:35433)
0926 04:07:51.533407 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:51.533410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:51.533625 (+ 215us) server_negotiation.cc:407] Connection header received
0926 04:07:51.534147 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:51.534149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:51.534160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:51.534222 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:51.535332 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.535907 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.537002 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.537214 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.537281 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.540040 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:51.540042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:51.540043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:51.540051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:51.541476 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.541985 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.541986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.541986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.542042 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.542319 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.542320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.542320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.542450 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:51.542529 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:51.542841 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 04:07:51.542936 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 04:07:51.543246 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.533303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35433 (local address 127.0.213.195:47853)
0926 04:07:51.533496 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:51.533534 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:51.534031 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:51.534316 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:51.534319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:51.535010 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:51.535216 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:51.535220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.536016 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.536016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:51.536899 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:51.536901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.537296 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.537296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:51.537401 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.538054 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:51.538062 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:51.539938 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:51.541571 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:51.541580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:51.541583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:51.541846 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:51.542116 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:51.542117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:51.542118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:51.542246 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:51.542519 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:51.542525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:51.542690 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 04:07:51.542808 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 04:07:51.542988 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260926 04:07:51.543751 4359 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35433
I20260926 04:07:51.543859 4359 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:51.544011 4359 heartbeater.cc:511] Master 127.0.213.254:35433 requested a full tablet report, sending...
I20260926 04:07:51.544507 3900 ts_manager.cc:194] Registered new tserver with Master: d26521e44bf74fb49314babac284db14 (127.0.213.195:42387)
I20260926 04:07:51.544873 3900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:47853
I20260926 04:07:51.551345 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[3855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395671, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:51.575639 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:51.567551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.213.254:35433)
0926 04:07:51.567646 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:51.567649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:51.567758 (+ 109us) server_negotiation.cc:407] Connection header received
0926 04:07:51.567849 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:51.567852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:51.567866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:51.567939 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:51.568684 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.569177 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.569934 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:51.570130 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:51.570217 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:51.572788 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:51.572790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:51.572790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:51.572799 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:51.574257 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.574652 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.574653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.574654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.574714 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:51.574965 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:51.574966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:51.574966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:51.575130 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:51.575240 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:51.575417 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 04:07:51.575504 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 04:07:51.575907 3900 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48348
I20260926 04:07:51.576450 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3962
I20260926 04:07:51.580969 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4096
I20260926 04:07:51.586054 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4230
I20260926 04:07:51.590847 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 3868
2026-09-26T04:07:51Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1937 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[4371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[4371](info): set up 2 sockets
Sep 26 04:07:51 dist-test-slave-xf24 krb5kdc[4371](info): commencing operation
krb5kdc: starting...
W20260926 04:07:52.636152 855 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:52 dist-test-slave-xf24 krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:52.768993 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41887
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:41887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:52.851759 4384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:52.851984 4384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:52.852007 4384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:52.853250 4384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:52.853296 4384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:52.853312 4384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:52.853323 4384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:52.853336 4384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:52.854759 4384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:41887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:52.854971 4384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:52.855144 4384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:07:52.860620 4384 server_base.cc:1034] running on GCE node
W20260926 04:07:52.860579 4390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:52.860587 4389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:52.860579 4392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:52.861189 4384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:52.861531 4384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:52.862718 4384 hybrid_clock.cc:648] HybridClock initialized: now 1790395672862695 us; error 43 us; skew 500 ppm
Sep 26 04:07:52 dist-test-slave-xf24 krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:52.865487 4384 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:52.866050 4384 webserver.cc:460] Webserver started at http://127.0.213.254:42303/ using document root <none> and password file <none>
I20260926 04:07:52.866520 4384 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:52.866611 4384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:52.866775 4384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:52.867967 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "f803b142e0ea4c239b2cb8f9e2208c57"
format_stamp: "Formatted at 2026-09-26 04:07:52 on dist-test-slave-xf24"
server_key: "bd6092cd7befdbb6a237f20c02de6409"
server_key_iv: "f3f7a5b5cbddee68b28e3f0d43a558ea"
server_key_version: "encryptionkey@0"
I20260926 04:07:52.868438 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "f803b142e0ea4c239b2cb8f9e2208c57"
format_stamp: "Formatted at 2026-09-26 04:07:52 on dist-test-slave-xf24"
server_key: "bd6092cd7befdbb6a237f20c02de6409"
server_key_iv: "f3f7a5b5cbddee68b28e3f0d43a558ea"
server_key_version: "encryptionkey@0"
I20260926 04:07:52.869907 4384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:52.870860 4399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:52.871174 4384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:52.871327 4384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "f803b142e0ea4c239b2cb8f9e2208c57"
format_stamp: "Formatted at 2026-09-26 04:07:52 on dist-test-slave-xf24"
server_key: "bd6092cd7befdbb6a237f20c02de6409"
server_key_iv: "f3f7a5b5cbddee68b28e3f0d43a558ea"
server_key_version: "encryptionkey@0"
I20260926 04:07:52.871428 4384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:52.894794 4384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:52.897703 4384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:52.897915 4384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:52.902649 4384 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:41887
I20260926 04:07:52.902685 4451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:41887 every 8 connection(s)
I20260926 04:07:52.903105 4384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:52.904078 4452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:52.904626 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4384
I20260926 04:07:52.904753 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:52.904934 855 external_mini_cluster.cc:1428] Setting key 974ab8e751c5f19c881dd82628f44e23
I20260926 04:07:52.907352 4452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57: Bootstrap starting.
I20260926 04:07:52.908004 4452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:52.908397 4452 log.cc:824] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:52.909204 4452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57: No bootstrap required, opened a new log
I20260926 04:07:52.911860 4452 raft_consensus.cc:348] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41887 } }
I20260926 04:07:52.912014 4452 raft_consensus.cc:374] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:52.912036 4452 raft_consensus.cc:729] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f803b142e0ea4c239b2cb8f9e2208c57, State: Initialized, Role: FOLLOWER
Sep 26 04:07:52 dist-test-slave-xf24 krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:52.912221 4452 consensus_queue.cc:260] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41887 } }
I20260926 04:07:52.912300 4452 raft_consensus.cc:388] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:52.912334 4452 raft_consensus.cc:482] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:52.912392 4452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:52.913169 4452 raft_consensus.cc:504] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41887 } }
I20260926 04:07:52.913293 4452 leader_election.cc:302] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f803b142e0ea4c239b2cb8f9e2208c57; no voters:
I20260926 04:07:52.913498 4452 leader_election.cc:288] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:52.913554 4457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:52.913806 4457 raft_consensus.cc:686] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [term 1 LEADER]: Becoming Leader. State: Replica: f803b142e0ea4c239b2cb8f9e2208c57, State: Running, Role: LEADER
I20260926 04:07:52.913833 4452 sys_catalog.cc:564] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:52.913980 4457 consensus_queue.cc:237] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41887 } }
I20260926 04:07:52.914381 4458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41887 } } }
I20260926 04:07:52.914485 4458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:52.914731 4465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:52.915274 4465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:52.915812 4459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f803b142e0ea4c239b2cb8f9e2208c57. Latest consensus state: current_term: 1 leader_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f803b142e0ea4c239b2cb8f9e2208c57" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41887 } } }
I20260926 04:07:52.915898 4459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:52.917260 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:52.905534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.213.254:41887)
0926 04:07:52.905722 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:52.905736 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:52.905773 (+ 37us) server_negotiation.cc:407] Connection header received
0926 04:07:52.906195 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:52.906202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:52.906266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:52.906391 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:52.907170 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:52.908209 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:52.909276 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:52.909700 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:52.909796 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:52.912840 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:52.912854 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:52.912854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:52.912866 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:52.915120 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:52.915589 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:52.915593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:52.915594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:52.915697 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:52.915967 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:52.915969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:52.915969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:52.916338 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:52.916488 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:52.916946 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 04:07:52.917071 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260926 04:07:52.917263 4465 catalog_manager.cc:1347] Generated new cluster ID: 60570c39e9a34bffbd70e5930691fb03
I20260926 04:07:52.917327 4465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:52.929220 4465 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:52.929998 4465 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:52.934139 4465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f803b142e0ea4c239b2cb8f9e2208c57: Generated new TSK 0
I20260926 04:07:52.934381 4465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:52.972529 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41887
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:53.065816 4478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:53.066397 4478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:53.066473 4478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:53.068609 4478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:53.068704 4478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:53.068764 4478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:53.070766 4478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:53.071127 4478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:53.071436 4478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:53.077459 4486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:53.077473 4484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:53.077471 4483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:53.077594 4478 server_base.cc:1034] running on GCE node
I20260926 04:07:53.078058 4478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:53.078320 4478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:53.079487 4478 hybrid_clock.cc:648] HybridClock initialized: now 1790395673079457 us; error 58 us; skew 500 ppm
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:53.082508 4478 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:53.083187 4478 webserver.cc:460] Webserver started at http://127.0.213.193:45307/ using document root <none> and password file <none>
I20260926 04:07:53.083420 4478 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:53.083465 4478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:53.083564 4478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:53.084741 4478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "72106791c2d842afa6faed5da5208134"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "690bff215d69ef654e8a256544f50b07"
server_key_iv: "d4599c4aa721f5c88420807f57a0bce3"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.085073 4478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "72106791c2d842afa6faed5da5208134"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "690bff215d69ef654e8a256544f50b07"
server_key_iv: "d4599c4aa721f5c88420807f57a0bce3"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.086570 4478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:53.087484 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.087785 4478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:53.087925 4478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "72106791c2d842afa6faed5da5208134"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "690bff215d69ef654e8a256544f50b07"
server_key_iv: "d4599c4aa721f5c88420807f57a0bce3"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.088022 4478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:53.121035 4478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:53.124035 4478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:53.124277 4478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:53.124588 4478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:53.124939 4478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:53.125000 4478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.125036 4478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:53.125080 4478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.131853 4478 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:46177
I20260926 04:07:53.131897 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:46177 every 8 connection(s)
I20260926 04:07:53.132292 4478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:53.139204 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4478
I20260926 04:07:53.139384 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:53.139626 855 external_mini_cluster.cc:1428] Setting key 4321d50b7743c54f64a00f4f6edf212d
I20260926 04:07:53.144683 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.133202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:36693 (local address 127.0.213.254:41887)
0926 04:07:53.133271 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:53.133274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:53.133465 (+ 191us) server_negotiation.cc:407] Connection header received
0926 04:07:53.134509 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:53.134510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:53.134522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:53.134603 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:53.135339 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.135863 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.136972 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.137198 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.137294 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.140419 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:53.140421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:53.140421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:53.140430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:53.142592 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.143354 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.143355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.143356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.143424 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.143814 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.143815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.143816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.143953 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:53.144015 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:53.144450 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 04:07:53.144549 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:07:53.144932 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.133098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41887 (local address 127.0.213.193:36693)
0926 04:07:53.133276 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:53.133326 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:53.134352 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:53.134704 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:53.134709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:53.135003 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:53.135209 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:53.135214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.135958 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.135959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:53.136855 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:53.136858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.137339 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.137340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:53.137461 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.138121 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:53.138130 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:53.140299 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:53.142800 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:53.142810 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:53.142812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:53.143257 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:53.143529 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:53.143530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:53.143530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:53.143692 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:53.144031 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:53.144037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:53.144281 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 04:07:53.144406 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 04:07:53.144607 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260926 04:07:53.145790 4607 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41887
I20260926 04:07:53.145900 4607 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:53.146117 4607 heartbeater.cc:511] Master 127.0.213.254:41887 requested a full tablet report, sending...
I20260926 04:07:53.146749 4416 ts_manager.cc:194] Registered new tserver with Master: 72106791c2d842afa6faed5da5208134 (127.0.213.193:46177)
I20260926 04:07:53.147886 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:36693
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:53.172000 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41887
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:53.256254 4612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:53.256501 4612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:53.256568 4612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:53.257947 4612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:53.258001 4612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:53.258033 4612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:53.259761 4612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:53.259981 4612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:53.260279 4612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:53.265852 4618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:53.265852 4617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:53.265852 4620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:53.266173 4612 server_base.cc:1034] running on GCE node
I20260926 04:07:53.266455 4612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:53.266749 4612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:53.267941 4612 hybrid_clock.cc:648] HybridClock initialized: now 1790395673267909 us; error 47 us; skew 500 ppm
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:53.271178 4612 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:53.271862 4612 webserver.cc:460] Webserver started at http://127.0.213.194:35517/ using document root <none> and password file <none>
I20260926 04:07:53.272179 4612 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:53.272253 4612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:53.272404 4612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:53.273660 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "73acdaf355d148b4adf0c3e1fc11f43f"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "c06676fc951cd32a2c5ffa2c07419d76"
server_key_iv: "3038b5d2fa757c96cb134a72a00ad2a9"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.274067 4612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "73acdaf355d148b4adf0c3e1fc11f43f"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "c06676fc951cd32a2c5ffa2c07419d76"
server_key_iv: "3038b5d2fa757c96cb134a72a00ad2a9"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.275664 4612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:53.276798 4627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.277043 4612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:53.277489 4612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "73acdaf355d148b4adf0c3e1fc11f43f"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "c06676fc951cd32a2c5ffa2c07419d76"
server_key_iv: "3038b5d2fa757c96cb134a72a00ad2a9"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.277595 4612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:53.307372 4612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:53.310493 4612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:53.310707 4612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:53.310992 4612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:53.311362 4612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:53.311424 4612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.311471 4612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:53.311522 4612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.319051 4612 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:46279
I20260926 04:07:53.319092 4740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:46279 every 8 connection(s)
I20260926 04:07:53.319464 4612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:53.328801 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4612
I20260926 04:07:53.328959 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:53.329217 855 external_mini_cluster.cc:1428] Setting key ea4c5cd6bf36f9000675d0062d6bb75c
I20260926 04:07:53.331190 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.320313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:38627 (local address 127.0.213.254:41887)
0926 04:07:53.320424 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:53.320427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:53.320659 (+ 232us) server_negotiation.cc:407] Connection header received
0926 04:07:53.321824 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:53.321825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:53.321839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:53.321900 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:53.322609 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.323140 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.324254 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.324471 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.324555 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.327500 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:53.327502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:53.327502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:53.327512 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:53.329464 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.330081 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.330083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.330083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.330160 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.330458 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.330459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.330460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.330592 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:53.330671 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:53.331001 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 04:07:53.331094 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 04:07:53.331429 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.320302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41887 (local address 127.0.213.194:38627)
0926 04:07:53.320504 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:53.320546 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:53.321690 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:53.321983 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:53.321987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:53.322275 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:53.322484 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:53.322489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.323273 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.323274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:53.324150 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:53.324153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.324563 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.324564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:53.324708 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.325339 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:53.325348 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:53.327355 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:53.329589 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:53.329600 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:53.329603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:53.329974 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:53.330246 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:53.330246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:53.330247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:53.330386 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:53.330675 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:53.330681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:53.330855 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 04:07:53.330968 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 04:07:53.331151 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260926 04:07:53.332123 4741 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41887
I20260926 04:07:53.332249 4741 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:53.332455 4741 heartbeater.cc:511] Master 127.0.213.254:41887 requested a full tablet report, sending...
I20260926 04:07:53.332906 4416 ts_manager.cc:194] Registered new tserver with Master: 73acdaf355d148b4adf0c3e1fc11f43f (127.0.213.194:46279)
I20260926 04:07:53.333353 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:38627
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:53.359899 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41887
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:53.450718 4746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:53.450984 4746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:53.451027 4746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:53.452404 4746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:53.452476 4746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:53.452521 4746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:53.454164 4746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:53.454469 4746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:53.454728 4746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:53.460361 4754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:53.460350 4752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:53.460392 4746 server_base.cc:1034] running on GCE node
W20260926 04:07:53.460352 4751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:53.460819 4746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:53.461133 4746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:53.462329 4746 hybrid_clock.cc:648] HybridClock initialized: now 1790395673462311 us; error 43 us; skew 500 ppm
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:53.465248 4746 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:53.465742 4746 webserver.cc:460] Webserver started at http://127.0.213.195:37517/ using document root <none> and password file <none>
I20260926 04:07:53.466010 4746 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:53.466058 4746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:53.466197 4746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:53.467296 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "19bbdf5eede9429ab454edfcb8d7341f"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "bd03ab80d5b2e92e4d49fc0f0b9eb6b9"
server_key_iv: "ea72e14760c65e9558c80c7b198d1c0e"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.467665 4746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "19bbdf5eede9429ab454edfcb8d7341f"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "bd03ab80d5b2e92e4d49fc0f0b9eb6b9"
server_key_iv: "ea72e14760c65e9558c80c7b198d1c0e"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.469310 4746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:53.470381 4761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.470616 4746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:53.470737 4746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "19bbdf5eede9429ab454edfcb8d7341f"
format_stamp: "Formatted at 2026-09-26 04:07:53 on dist-test-slave-xf24"
server_key: "bd03ab80d5b2e92e4d49fc0f0b9eb6b9"
server_key_iv: "ea72e14760c65e9558c80c7b198d1c0e"
server_key_version: "encryptionkey@0"
I20260926 04:07:53.470803 4746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:53.488687 4746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:53.490809 4746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:53.490947 4746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:53.491182 4746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:53.491508 4746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:53.491566 4746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.491609 4746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:53.491634 4746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:53.497438 4746 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:41153
I20260926 04:07:53.497522 4874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:41153 every 8 connection(s)
I20260926 04:07:53.497867 4746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:53.505868 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4746
I20260926 04:07:53.505941 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:53.506170 855 external_mini_cluster.cc:1428] Setting key 972981aaff98c3046763d62521b49c93
I20260926 04:07:53.508924 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.498770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:56147 (local address 127.0.213.254:41887)
0926 04:07:53.498856 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:53.498858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:53.499166 (+ 308us) server_negotiation.cc:407] Connection header received
0926 04:07:53.500313 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:53.500315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:53.500327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:53.500394 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:53.501091 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.501606 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.502636 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.502840 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.502901 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.505364 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:53.505366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:53.505367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:53.505374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:53.507329 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.507792 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.507793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.507793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.507853 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.508274 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.508275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.508275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.508399 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:53.508475 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:53.508781 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 04:07:53.508856 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:07:53.509083 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.498788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41887 (local address 127.0.213.195:56147)
0926 04:07:53.498979 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:53.499048 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:53.500189 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:53.500456 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:53.500460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:53.500764 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:53.500965 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:53.500969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.501697 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.501698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:53.502530 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:53.502533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.502956 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.502958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:53.503075 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.503662 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:53.503671 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:53.505197 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:53.507423 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:53.507431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:53.507433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:53.507702 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:53.507963 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:53.507963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:53.507964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:53.508180 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:53.508471 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:53.508476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:53.508612 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 04:07:53.508697 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 04:07:53.508867 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260926 04:07:53.509553 4875 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41887
I20260926 04:07:53.509665 4875 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:53.509879 4875 heartbeater.cc:511] Master 127.0.213.254:41887 requested a full tablet report, sending...
I20260926 04:07:53.510388 4416 ts_manager.cc:194] Registered new tserver with Master: 19bbdf5eede9429ab454edfcb8d7341f (127.0.213.195:41153)
I20260926 04:07:53.510814 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:56147
I20260926 04:07:53.518431 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:53.542637 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.534559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.213.193:46177)
0926 04:07:53.534754 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:53.534759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:53.534771 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:07:53.534797 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:53.534798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:53.534812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:53.534882 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:53.535468 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.536338 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.537041 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.537369 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.537441 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.539703 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:53.539712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:53.539713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:53.539720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:53.541389 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.541779 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.541782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.541782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.541842 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.542104 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.542105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.542105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.542255 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:53.542384 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:53.542482 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 04:07:53.542545 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:53.552577 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:53.544405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.0.213.254:41887)
0926 04:07:53.544474 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:53.544476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:53.544485 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:53.544562 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:53.544563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:53.544572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:53.544611 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:53.545269 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.545774 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.546494 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:53.546691 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:53.546758 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:53.549380 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:53.549382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:53.549383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:53.549392 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:53.551098 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.551562 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.551563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.551564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.551636 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:53.551869 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:53.551870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:53.551870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:53.552048 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:53.552207 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:53.552362 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 04:07:53.552454 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 04:07:53.552850 4416 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34624
I20260926 04:07:53.553256 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4478
I20260926 04:07:53.557618 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4612
I20260926 04:07:53.562577 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4746
I20260926 04:07:53.567044 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4384
2026-09-26T04:07:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1975 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4887](info): set up 2 sockets
Sep 26 04:07:53 dist-test-slave-xf24 krb5kdc[4887](info): commencing operation
krb5kdc: starting...
W20260926 04:07:54.624536 855 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:54 dist-test-slave-xf24 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:54.755784 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:33101
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:54.839203 4900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:54.839479 4900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:54.839519 4900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:54.840879 4900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:54.840973 4900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:54.841003 4900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:54.841037 4900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:54.841061 4900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:54.842703 4900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:33101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:54.843024 4900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:54.843278 4900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:54.848356 4906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:54.848356 4905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:54.848356 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:54.848609 4900 server_base.cc:1034] running on GCE node
I20260926 04:07:54.848881 4900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:54.849189 4900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:54.850402 4900 hybrid_clock.cc:648] HybridClock initialized: now 1790395674850386 us; error 45 us; skew 500 ppm
Sep 26 04:07:54 dist-test-slave-xf24 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:54.859704 4900 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:54.860183 4900 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:54.860239 4900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:54.860419 4900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:54.861579 4900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "0b6bc9d7e08f49868fc3ff3b9375008f"
format_stamp: "Formatted at 2026-09-26 04:07:54 on dist-test-slave-xf24"
server_key: "e5893bd3d6bfed50df76ed0437304002"
server_key_iv: "798aec5be73fb3d6256b17e8a1a3e8f1"
server_key_version: "encryptionkey@0"
I20260926 04:07:54.861995 4900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "0b6bc9d7e08f49868fc3ff3b9375008f"
format_stamp: "Formatted at 2026-09-26 04:07:54 on dist-test-slave-xf24"
server_key: "e5893bd3d6bfed50df76ed0437304002"
server_key_iv: "798aec5be73fb3d6256b17e8a1a3e8f1"
server_key_version: "encryptionkey@0"
I20260926 04:07:54.863377 4900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:54.864226 4914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:54.864408 4900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:54.864504 4900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "0b6bc9d7e08f49868fc3ff3b9375008f"
format_stamp: "Formatted at 2026-09-26 04:07:54 on dist-test-slave-xf24"
server_key: "e5893bd3d6bfed50df76ed0437304002"
server_key_iv: "798aec5be73fb3d6256b17e8a1a3e8f1"
server_key_version: "encryptionkey@0"
I20260926 04:07:54.864566 4900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:54.879724 4900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:54.881939 4900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:54.882074 4900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:54.886265 4900 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:33101
I20260926 04:07:54.886308 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:33101 every 8 connection(s)
I20260926 04:07:54.886633 4900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:54.887598 4967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:54.889990 4967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: Bootstrap starting.
I20260926 04:07:54.890628 4967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:54.890930 4967 log.cc:824] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:54.891290 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4900
I20260926 04:07:54.891409 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:54.891580 855 external_mini_cluster.cc:1428] Setting key cfa311f9fc95c77af55cc72e1d1a6a28
I20260926 04:07:54.891741 4967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: No bootstrap required, opened a new log
I20260926 04:07:54.893541 4967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 33101 } }
I20260926 04:07:54.893719 4967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:54.893782 4967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6bc9d7e08f49868fc3ff3b9375008f, State: Initialized, Role: FOLLOWER
I20260926 04:07:54.893929 4967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 33101 } }
I20260926 04:07:54.894032 4967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:54.894078 4967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:54.894129 4967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:54.895011 4967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 33101 } }
I20260926 04:07:54.895136 4967 leader_election.cc:302] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b6bc9d7e08f49868fc3ff3b9375008f; no voters:
I20260926 04:07:54.895320 4967 leader_election.cc:288] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:54.895414 4972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:54.895560 4972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [term 1 LEADER]: Becoming Leader. State: Replica: 0b6bc9d7e08f49868fc3ff3b9375008f, State: Running, Role: LEADER
I20260926 04:07:54.895584 4967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:54.895812 4972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 33101 } }
I20260926 04:07:54.896806 4974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 33101 } } }
I20260926 04:07:54.896898 4974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:54.897221 4973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b6bc9d7e08f49868fc3ff3b9375008f. Latest consensus state: current_term: 1 leader_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6bc9d7e08f49868fc3ff3b9375008f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 33101 } } }
I20260926 04:07:54.897500 4973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f [sys.catalog]: This master's current role is: LEADER
W20260926 04:07:54.897651 4987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:07:54.897729 4987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:07:54.897809 4986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:54.898032 4986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:54.899760 4986 catalog_manager.cc:1347] Generated new cluster ID: bbb9bc8c55a044688ae9c16429a867da
I20260926 04:07:54.899812 4986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:07:54 dist-test-slave-xf24 krb5kdc[4887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:54.905452 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:54.892148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.0.213.254:33101)
0926 04:07:54.892355 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:54.892366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:54.892397 (+ 31us) server_negotiation.cc:407] Connection header received
0926 04:07:54.893554 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:54.893563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:54.893625 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:54.893796 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:54.894522 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:54.895703 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:54.896882 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:54.897532 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:54.897666 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:54.900926 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:54.900930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:54.900931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:54.900944 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:54.903392 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:54.903813 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:54.903817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:54.903817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:54.903927 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:54.904178 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:54.904178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:54.904179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:54.904475 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:54.904611 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:54.905069 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 04:07:54.905205 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260926 04:07:54.919050 4986 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:54.919836 4986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:54.923180 4986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: Generated new TSK 0
I20260926 04:07:54.923378 4986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:54.955255 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:33101
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:55.043111 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:55.043357 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:55.043385 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:55.044728 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:55.044778 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:55.044811 4993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:55.046478 4993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:55.046761 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:55.047034 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:55.052471 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.052454 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.052480 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.052932 4993 server_base.cc:1034] running on GCE node
I20260926 04:07:55.053143 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:55.053364 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:55.054514 4993 hybrid_clock.cc:648] HybridClock initialized: now 1790395675054506 us; error 35 us; skew 500 ppm
Sep 26 04:07:55 dist-test-slave-xf24 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:55.057192 4993 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:55.057639 4993 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:55.057720 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:55.057878 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:55.058998 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "604f357427c54975b5c48543b08d0b87"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "e757b19c6268e2db0734b0c50cd85f20"
server_key_iv: "431f8277a815927a232207369399214d"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.059374 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "604f357427c54975b5c48543b08d0b87"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "e757b19c6268e2db0734b0c50cd85f20"
server_key_iv: "431f8277a815927a232207369399214d"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.060863 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:55.061587 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.061755 4993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:07:55.061862 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "604f357427c54975b5c48543b08d0b87"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "e757b19c6268e2db0734b0c50cd85f20"
server_key_iv: "431f8277a815927a232207369399214d"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.061939 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:55.099324 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:55.102051 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:55.102248 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:55.102571 4993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:55.102993 4993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:55.103055 4993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.103080 4993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:55.103093 4993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.110319 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:46705
I20260926 04:07:55.110411 5120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:46705 every 8 connection(s)
I20260926 04:07:55.110708 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:55.111155 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 4993
I20260926 04:07:55.111271 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:55.111495 855 external_mini_cluster.cc:1428] Setting key cd7d9bb64842c8f12d1e9aef26f2750a
Sep 26 04:07:55 dist-test-slave-xf24 krb5kdc[4887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:07:55.122584 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.111698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:49133 (local address 127.0.213.254:33101)
0926 04:07:55.111757 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.111760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.112124 (+ 364us) server_negotiation.cc:407] Connection header received
0926 04:07:55.112936 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.112938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.112957 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.113016 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:55.114293 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.115141 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.116357 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.116736 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.116828 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.119196 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:55.119198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:55.119198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:55.119209 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:55.120815 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.121470 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.121471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.121472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.121543 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.121860 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.121862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.121862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.122009 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:55.122074 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.122420 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.122509 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 04:07:55.122798 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.111696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:33101 (local address 127.0.213.193:49133)
0926 04:07:55.111976 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.112020 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.112791 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.113113 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.113118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.113556 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:55.114176 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.114182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.115245 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.115246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.116265 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.116268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.116859 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.116860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.117002 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.117644 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:55.117653 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:55.119080 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:55.120918 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:55.120962 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:55.120965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:55.121335 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:55.121635 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:55.121636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:55.121636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:55.121761 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:55.122072 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:55.122085 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:55.122247 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.122344 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.122509 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
I20260926 04:07:55.123320 5121 heartbeater.cc:347] Connected to a master server at 127.0.213.254:33101
I20260926 04:07:55.123459 5121 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:55.123651 5121 heartbeater.cc:511] Master 127.0.213.254:33101 requested a full tablet report, sending...
I20260926 04:07:55.124279 4931 ts_manager.cc:194] Registered new tserver with Master: 604f357427c54975b5c48543b08d0b87 (127.0.213.193:46705)
I20260926 04:07:55.125082 4931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:49133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:55.143422 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:33101
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:55.228001 5126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:55.228262 5126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:55.228292 5126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:55.229621 5126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:55.229681 5126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:55.229712 5126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:55.231426 5126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:55.231719 5126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:55.231994 5126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:55.237254 5131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.237254 5132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.237254 5134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.237558 5126 server_base.cc:1034] running on GCE node
I20260926 04:07:55.237949 5126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:55.238260 5126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:55.239435 5126 hybrid_clock.cc:648] HybridClock initialized: now 1790395675239421 us; error 29 us; skew 500 ppm
Sep 26 04:07:55 dist-test-slave-xf24 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:55.242262 5126 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:55.242650 5126 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:55.242726 5126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:55.242892 5126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:55.244020 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "fa25fec350594ce8b9ab410b135db81e"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "43a37970a046da788acbcf440ab1cce0"
server_key_iv: "977db3d4b0d49c3911c8ed1cf2bc7f35"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.244477 5126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "fa25fec350594ce8b9ab410b135db81e"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "43a37970a046da788acbcf440ab1cce0"
server_key_iv: "977db3d4b0d49c3911c8ed1cf2bc7f35"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.245967 5126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:07:55.246855 5140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.247067 5126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:55.247179 5126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "fa25fec350594ce8b9ab410b135db81e"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "43a37970a046da788acbcf440ab1cce0"
server_key_iv: "977db3d4b0d49c3911c8ed1cf2bc7f35"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.247257 5126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:55.256112 5126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:55.258656 5126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:55.258841 5126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:55.259093 5126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:55.259421 5126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:55.259476 5126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.259521 5126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:55.259554 5126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.265552 5126 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:34031
I20260926 04:07:55.265580 5253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:34031 every 8 connection(s)
I20260926 04:07:55.265872 5126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:55.268493 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5126
I20260926 04:07:55.268592 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:55.268786 855 external_mini_cluster.cc:1428] Setting key 6989535a8a6cf052a0e1e56e209be6ca
Sep 26 04:07:55 dist-test-slave-xf24 krb5kdc[4887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:55.276453 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.266919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:34033 (local address 127.0.213.254:33101)
0926 04:07:55.266990 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.266993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.267205 (+ 212us) server_negotiation.cc:407] Connection header received
0926 04:07:55.267741 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.267742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.267753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.267799 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:55.268970 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.269516 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.270627 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.270849 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.270941 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.273334 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:55.273335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:55.273336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:55.273344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:55.274883 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.275379 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.275380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.275380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.275435 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.275697 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.275697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.275698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.275826 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:55.275900 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.276255 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.276356 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:07:55.276666 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.266913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:33101 (local address 127.0.213.194:34033)
0926 04:07:55.267071 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.267109 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.267598 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.267887 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.267890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.268242 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:55.268860 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.268868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.269622 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.269624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.270483 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.270486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.270954 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.270955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.271097 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.271708 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:55.271717 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:55.273221 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:55.274998 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:55.275006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:55.275008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:55.275244 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:55.275510 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:55.275511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:55.275511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:55.275621 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:55.275905 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:55.275911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:55.276101 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.276235 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.276407 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:07:55.277187 5254 heartbeater.cc:347] Connected to a master server at 127.0.213.254:33101
I20260926 04:07:55.277287 5254 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:55.277467 5254 heartbeater.cc:511] Master 127.0.213.254:33101 requested a full tablet report, sending...
I20260926 04:07:55.277869 4931 ts_manager.cc:194] Registered new tserver with Master: fa25fec350594ce8b9ab410b135db81e (127.0.213.194:34031)
I20260926 04:07:55.278307 4931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:34033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:55.295848 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:33101
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:55.378736 5259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:55.378957 5259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:55.378981 5259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:55.380167 5259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:55.380228 5259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:55.380257 5259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:55.382092 5259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:33101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:55.382314 5259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:55.382519 5259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:55.387583 5265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.387583 5267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.387635 5264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.387948 5259 server_base.cc:1034] running on GCE node
I20260926 04:07:55.388216 5259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:55.388474 5259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:55.389617 5259 hybrid_clock.cc:648] HybridClock initialized: now 1790395675389607 us; error 30 us; skew 500 ppm
Sep 26 04:07:55 dist-test-slave-xf24 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:55.392278 5259 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:55.392695 5259 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:55.392745 5259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:55.392834 5259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:55.393850 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "9990c274b5704cadb0421af437fe46e3"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "7bc42ff6b89fad3039eba87f91f5ed6e"
server_key_iv: "02b5a34a0b671e9a19d194ee9fa3efdf"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.394152 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "9990c274b5704cadb0421af437fe46e3"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "7bc42ff6b89fad3039eba87f91f5ed6e"
server_key_iv: "02b5a34a0b671e9a19d194ee9fa3efdf"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.395356 5259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:07:55.396337 5273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.396597 5259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:55.396701 5259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "9990c274b5704cadb0421af437fe46e3"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "7bc42ff6b89fad3039eba87f91f5ed6e"
server_key_iv: "02b5a34a0b671e9a19d194ee9fa3efdf"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.396783 5259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:55.430476 5259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:55.432958 5259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:55.433130 5259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:55.433418 5259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:55.433740 5259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:55.433794 5259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.433837 5259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:55.433862 5259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.439530 5259 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:43251
I20260926 04:07:55.439561 5386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:43251 every 8 connection(s)
I20260926 04:07:55.439854 5259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:55.441577 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5259
I20260926 04:07:55.441680 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:55.441859 855 external_mini_cluster.cc:1428] Setting key 51ee05dc92b5871a13c18255bbdfc744
Sep 26 04:07:55 dist-test-slave-xf24 krb5kdc[4887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:55.449546 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.440618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:41191 (local address 127.0.213.254:33101)
0926 04:07:55.440672 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.440674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.440871 (+ 197us) server_negotiation.cc:407] Connection header received
0926 04:07:55.441412 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.441413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.441425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.441492 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:55.442788 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.443305 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.444411 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.444607 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.444668 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.446886 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:55.446888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:55.446888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:55.446895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:55.448340 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.448729 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.448731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.448731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.448793 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.449017 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.449018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.449018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.449126 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:55.449178 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.449418 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.449479 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:07:55.449783 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.440570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:33101 (local address 127.0.213.195:41191)
0926 04:07:55.440728 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.440770 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.441254 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.441600 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.441605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.442017 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:55.442703 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.442709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.443425 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.443426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.444325 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.444328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.444684 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.444685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.444782 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.445407 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:55.445415 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:55.446776 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:55.448410 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:55.448416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:55.448418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:55.448635 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:55.448860 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:55.448861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:55.448861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:55.448960 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:55.449181 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:55.449185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:55.449303 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.449393 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.449516 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260926 04:07:55.450341 5387 heartbeater.cc:347] Connected to a master server at 127.0.213.254:33101
I20260926 04:07:55.450440 5387 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:55.450620 5387 heartbeater.cc:511] Master 127.0.213.254:33101 requested a full tablet report, sending...
I20260926 04:07:55.451038 4931 ts_manager.cc:194] Registered new tserver with Master: 9990c274b5704cadb0421af437fe46e3 (127.0.213.195:43251)
I20260926 04:07:55.451404 4931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:41191
I20260926 04:07:55.453634 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:07:55.461071 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.455026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.0.213.254:33101)
0926 04:07:55.455093 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.455094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.455197 (+ 103us) server_negotiation.cc:407] Connection header received
0926 04:07:55.455233 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.455233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.455242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.455294 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:55.455815 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.456353 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.457029 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.457210 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.457266 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.458412 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:55.458413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:55.458413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:55.458420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:55.459798 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.460153 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.460153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.460154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.460216 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:55.460485 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:55.460487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:55.460487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:55.460666 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:55.460799 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.460937 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.461001 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:07:55.463165 4931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:55.465013 4931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:55.472239 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.467538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.0.213.195:43251)
0926 04:07:55.467716 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.467723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.467737 (+ 14us) server_negotiation.cc:407] Connection header received
0926 04:07:55.467797 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.467799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.467827 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.467910 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:55.468224 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.469395 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.471043 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.471711 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.471780 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.471839 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.472094 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.472139 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":117,"threads_started":1}
I20260926 04:07:55.472263 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.467457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43251 (local address 127.0.0.1:56332)
0926 04:07:55.467576 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.467617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.467682 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.468017 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.468021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.468023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:55.468125 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.468127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.469527 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.469528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.470948 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.470951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.471821 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.471821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.471932 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.471946 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.472035 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.472158 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":121,"thread_start_us":71,"threads_started":1}
I20260926 04:07:55.472538 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.467975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.0.213.193:46705)
0926 04:07:55.468358 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.468363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.468372 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:55.468409 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.468411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.468426 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.468504 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:55.468728 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.469722 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.470885 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.471677 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.471770 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.471840 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.472412 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.472458 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":70,"threads_started":1}
I20260926 04:07:55.472263 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.467863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:46705 (local address 127.0.0.1:60920)
0926 04:07:55.467966 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.467974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.468045 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.468596 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.468596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.468599 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:55.468639 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.468640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.469842 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.469843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.470784 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.470787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.471820 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.471821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.471927 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.471946 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.472035 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.472146 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260926 04:07:55.473214 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.467666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34031 (local address 127.0.0.1:40928)
0926 04:07:55.467801 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.467812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.467868 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.468484 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.468485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.468487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:55.468540 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.468542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.469525 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.469527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.470952 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.470953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.472785 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.472786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.472864 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.472869 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.472903 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.472952 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":52,"thread_start_us":79,"threads_started":1}
I20260926 04:07:55.473198 5321 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8aa5b53131480692fd348e753341eb (DEFAULT_TABLE table=test-table [id=358afd27e10440d8ba0a2dcd30248233]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:55.473593 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8aa5b53131480692fd348e753341eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:55.473623 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.468128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.0.213.194:34031)
0926 04:07:55.468268 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.468272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.468280 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:55.468315 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.468317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.468333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.468613 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:55.468695 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.469384 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.471884 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.472673 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.473416 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.473503 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.473527 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.473562 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260926 04:07:55.473598 5055 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8aa5b53131480692fd348e753341eb (DEFAULT_TABLE table=test-table [id=358afd27e10440d8ba0a2dcd30248233]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:55.473912 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8aa5b53131480692fd348e753341eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:55.474144 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8aa5b53131480692fd348e753341eb (DEFAULT_TABLE table=test-table [id=358afd27e10440d8ba0a2dcd30248233]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:55.474376 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8aa5b53131480692fd348e753341eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:55.477152 5405 tablet_bootstrap.cc:492] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: Bootstrap starting.
I20260926 04:07:55.477500 5406 tablet_bootstrap.cc:492] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Bootstrap starting.
I20260926 04:07:55.477823 5405 tablet_bootstrap.cc:654] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:55.478178 5406 tablet_bootstrap.cc:654] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:55.478308 5405 log.cc:824] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:55.478643 5404 tablet_bootstrap.cc:492] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Bootstrap starting.
I20260926 04:07:55.478881 5406 log.cc:824] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:55.479161 5405 tablet_bootstrap.cc:492] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: No bootstrap required, opened a new log
I20260926 04:07:55.479290 5405 ts_tablet_manager.cc:1397] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:55.479526 5404 tablet_bootstrap.cc:654] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:55.479614 5406 tablet_bootstrap.cc:492] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: No bootstrap required, opened a new log
I20260926 04:07:55.479688 5406 ts_tablet_manager.cc:1397] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:55.480154 5404 log.cc:824] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:55.480862 5404 tablet_bootstrap.cc:492] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: No bootstrap required, opened a new log
I20260926 04:07:55.480944 5404 ts_tablet_manager.cc:1397] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:07:55.481572 5405 raft_consensus.cc:348] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.481770 5405 raft_consensus.cc:374] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:55.481808 5405 raft_consensus.cc:729] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604f357427c54975b5c48543b08d0b87, State: Initialized, Role: FOLLOWER
I20260926 04:07:55.481760 5406 raft_consensus.cc:348] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.481920 5406 raft_consensus.cc:374] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:55.481956 5406 raft_consensus.cc:729] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa25fec350594ce8b9ab410b135db81e, State: Initialized, Role: FOLLOWER
I20260926 04:07:55.481972 5405 consensus_queue.cc:260] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.482051 5406 consensus_queue.cc:260] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.482275 5405 ts_tablet_manager.cc:1428] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:55.482286 5406 ts_tablet_manager.cc:1428] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:55.482293 5121 heartbeater.cc:503] Master 127.0.213.254:33101 was elected leader, sending a full tablet report...
I20260926 04:07:55.482443 5254 heartbeater.cc:503] Master 127.0.213.254:33101 was elected leader, sending a full tablet report...
I20260926 04:07:55.482752 5404 raft_consensus.cc:348] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.482933 5404 raft_consensus.cc:374] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:55.482977 5404 raft_consensus.cc:729] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9990c274b5704cadb0421af437fe46e3, State: Initialized, Role: FOLLOWER
I20260926 04:07:55.483112 5404 consensus_queue.cc:260] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.483414 5404 ts_tablet_manager.cc:1428] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:07:55.483575 5387 heartbeater.cc:503] Master 127.0.213.254:33101 was elected leader, sending a full tablet report...
W20260926 04:07:55.516502 5255 tablet.cc:2369] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:07:55.611830 5122 tablet.cc:2369] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:07:55.635028 5412 raft_consensus.cc:482] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:55.635151 5412 raft_consensus.cc:504] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.635483 5412 leader_election.cc:288] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa25fec350594ce8b9ab410b135db81e (127.0.213.194:34031), 604f357427c54975b5c48543b08d0b87 (127.0.213.193:46705)
I20260926 04:07:55.639613 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.635961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:41757 (local address 127.0.213.193:46705)
0926 04:07:55.636025 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.636027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.636160 (+ 133us) server_negotiation.cc:407] Connection header received
0926 04:07:55.636216 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.636217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.636226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.636286 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:55.636423 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.637121 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.638603 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.639204 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.639287 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.639295 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.639497 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.639536 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:07:55.639613 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.635783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:46639 (local address 127.0.213.194:34031)
0926 04:07:55.635853 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.635855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.635870 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:07:55.635956 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.635957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.635966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.636018 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:07:55.636267 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.637029 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.638609 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.639211 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.639287 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.639295 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.639498 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.639536 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:07:55.639706 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.635888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:46705 (local address 127.0.213.195:41757)
0926 04:07:55.636087 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.636098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.636159 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.636306 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.636307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.636308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:55.636359 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.636360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.637197 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.637198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.638490 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.638493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.639326 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.639328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.639408 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.639412 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.639477 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.639538 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":77,"thread_start_us":118,"threads_started":1}
I20260926 04:07:55.639760 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.635713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34031 (local address 127.0.213.195:46639)
0926 04:07:55.635789 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.635816 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.635887 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.636031 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.636033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.636035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:07:55.636138 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.636140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.637129 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.637130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.638490 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:55.638493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.639349 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.639349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:55.639419 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.639421 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.639477 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.639529 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":11}
I20260926 04:07:55.640149 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8aa5b53131480692fd348e753341eb" candidate_uuid: "9990c274b5704cadb0421af437fe46e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604f357427c54975b5c48543b08d0b87" is_pre_election: true
I20260926 04:07:55.640177 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8aa5b53131480692fd348e753341eb" candidate_uuid: "9990c274b5704cadb0421af437fe46e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa25fec350594ce8b9ab410b135db81e" is_pre_election: true
I20260926 04:07:55.640331 5208 raft_consensus.cc:2454] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9990c274b5704cadb0421af437fe46e3 in term 0.
I20260926 04:07:55.640313 5075 raft_consensus.cc:2454] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9990c274b5704cadb0421af437fe46e3 in term 0.
I20260926 04:07:55.640589 5274 leader_election.cc:302] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604f357427c54975b5c48543b08d0b87, 9990c274b5704cadb0421af437fe46e3; no voters:
I20260926 04:07:55.640751 5412 raft_consensus.cc:2781] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:55.640873 5412 raft_consensus.cc:482] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:55.640913 5412 raft_consensus.cc:3037] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:55.641638 5412 raft_consensus.cc:504] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.641811 5412 leader_election.cc:288] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [CANDIDATE]: Term 1 election: Requested vote from peers fa25fec350594ce8b9ab410b135db81e (127.0.213.194:34031), 604f357427c54975b5c48543b08d0b87 (127.0.213.193:46705)
I20260926 04:07:55.642098 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8aa5b53131480692fd348e753341eb" candidate_uuid: "9990c274b5704cadb0421af437fe46e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa25fec350594ce8b9ab410b135db81e"
I20260926 04:07:55.642099 5075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8aa5b53131480692fd348e753341eb" candidate_uuid: "9990c274b5704cadb0421af437fe46e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604f357427c54975b5c48543b08d0b87"
I20260926 04:07:55.642198 5208 raft_consensus.cc:3037] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:55.642198 5075 raft_consensus.cc:3037] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:55.643134 5208 raft_consensus.cc:2454] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9990c274b5704cadb0421af437fe46e3 in term 1.
I20260926 04:07:55.643134 5075 raft_consensus.cc:2454] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9990c274b5704cadb0421af437fe46e3 in term 1.
I20260926 04:07:55.643368 5274 leader_election.cc:302] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604f357427c54975b5c48543b08d0b87, 9990c274b5704cadb0421af437fe46e3; no voters:
I20260926 04:07:55.643536 5412 raft_consensus.cc:2781] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:55.643705 5412 raft_consensus.cc:686] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 1 LEADER]: Becoming Leader. State: Replica: 9990c274b5704cadb0421af437fe46e3, State: Running, Role: LEADER
I20260926 04:07:55.643877 5412 consensus_queue.cc:237] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } }
I20260926 04:07:55.644917 4931 catalog_manager.cc:5495] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9990c274b5704cadb0421af437fe46e3 (127.0.213.195). New cstate: current_term: 1 leader_uuid: "9990c274b5704cadb0421af437fe46e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9990c274b5704cadb0421af437fe46e3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:55.654192 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.651680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.0.213.195:43251)
0926 04:07:55.651773 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.651775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.651785 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:55.651823 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.651824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.651840 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.651927 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:07:55.652089 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.652595 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.653386 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:55.653576 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:55.653655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:55.653822 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:07:55.653925 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:07:55.654026 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.654061 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.654081 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 04:07:55.656205 5075 raft_consensus.cc:1261] T 7f8aa5b53131480692fd348e753341eb P 604f357427c54975b5c48543b08d0b87 [term 1 FOLLOWER]: Refusing update from remote peer 9990c274b5704cadb0421af437fe46e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:55.656235 5208 raft_consensus.cc:1261] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 1 FOLLOWER]: Refusing update from remote peer 9990c274b5704cadb0421af437fe46e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:55.656592 5412 consensus_queue.cc:1035] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa25fec350594ce8b9ab410b135db81e" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:55.656765 5414 consensus_queue.cc:1035] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604f357427c54975b5c48543b08d0b87" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:55.659029 5417 mvcc.cc:205] Tried to move back new op lower bound from 7333460687484940288 to 7333460687438483456. Current Snapshot: MvccSnapshot[applied={T|T < 7333460687484940288}]
I20260926 04:07:55.659261 5418 mvcc.cc:205] Tried to move back new op lower bound from 7333460687484940288 to 7333460687438483456. Current Snapshot: MvccSnapshot[applied={T|T < 7333460687484940288}]
I20260926 04:07:55.661422 5419 mvcc.cc:205] Tried to move back new op lower bound from 7333460687484940288 to 7333460687438483456. Current Snapshot: MvccSnapshot[applied={T|T < 7333460687484940288 or (T in {7333460687484940288})}]
I20260926 04:07:55.661866 4930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:55.661967 4930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:55.662899 4930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b6bc9d7e08f49868fc3ff3b9375008f: Sending DeleteTablet for 3 replicas of tablet 7f8aa5b53131480692fd348e753341eb
I20260926 04:07:55.663303 5321 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8aa5b53131480692fd348e753341eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:55 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:56332
I20260926 04:07:55.663455 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8aa5b53131480692fd348e753341eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:55 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40928
I20260926 04:07:55.663566 5426 tablet_replica.cc:330] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: stopping tablet replica
I20260926 04:07:55.663676 5426 raft_consensus.cc:2229] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:55.663709 5427 tablet_replica.cc:330] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: stopping tablet replica
I20260926 04:07:55.663702 5055 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8aa5b53131480692fd348e753341eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:55 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:60920
I20260926 04:07:55.663851 5427 raft_consensus.cc:2229] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:55.663895 5426 raft_consensus.cc:2258] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:55.664003 5427 raft_consensus.cc:2258] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:55.664258 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4993
I20260926 04:07:55.665529 5426 ts_tablet_manager.cc:1905] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:55.665596 5427 ts_tablet_manager.cc:1905] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:55.667593 5427 ts_tablet_manager.cc:1918] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:55.667763 5427 log.cc:1196] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/7f8aa5b53131480692fd348e753341eb
I20260926 04:07:55.667968 5426 ts_tablet_manager.cc:1918] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:55.668143 5427 ts_tablet_manager.cc:1939] T 7f8aa5b53131480692fd348e753341eb P fa25fec350594ce8b9ab410b135db81e: Deleting consensus metadata
I20260926 04:07:55.668149 5426 log.cc:1196] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/7f8aa5b53131480692fd348e753341eb
I20260926 04:07:55.668502 5426 ts_tablet_manager.cc:1939] T 7f8aa5b53131480692fd348e753341eb P 9990c274b5704cadb0421af437fe46e3: Deleting consensus metadata
I20260926 04:07:55.668668 4915 catalog_manager.cc:4841] TS fa25fec350594ce8b9ab410b135db81e (127.0.213.194:34031): tablet 7f8aa5b53131480692fd348e753341eb (table test-table [id=358afd27e10440d8ba0a2dcd30248233]) successfully deleted
I20260926 04:07:55.669042 4915 catalog_manager.cc:4841] TS 9990c274b5704cadb0421af437fe46e3 (127.0.213.195:43251): tablet 7f8aa5b53131480692fd348e753341eb (table test-table [id=358afd27e10440d8ba0a2dcd30248233]) successfully deleted
W20260926 04:07:55.670871 4915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:46705 (error 108)
I20260926 04:07:55.671036 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5126
I20260926 04:07:55.672606 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.672464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:46705 (local address 127.0.0.1:60922)
0926 04:07:55.672524 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.672567 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 04:07:55.672734 4915 catalog_manager.cc:4584] TS 604f357427c54975b5c48543b08d0b87 (127.0.213.193:46705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f8aa5b53131480692fd348e753341eb: Network error: Client connection negotiation failed: client connection to 127.0.213.193:46705: connect: Connection refused (error 111)
I20260926 04:07:55.676048 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5259
I20260926 04:07:55.681875 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 4900
2026-09-26T04:07:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2115 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T04:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:55Z Disabled control of system clock
I20260926 04:07:55.711032 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35455
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:35455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 04:07:55.792279 5433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:55.792506 5433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:55.792528 5433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:55.793757 5433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:55.793812 5433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:55.793824 5433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:55.793838 5433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:55.793879 5433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:55.795341 5433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:35455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:55.795630 5433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:55.795969 5433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:55.801172 5438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.801249 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.801221 5433 server_base.cc:1034] running on GCE node
W20260926 04:07:55.801190 5439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.801692 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:55.802016 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:55.803182 5433 hybrid_clock.cc:648] HybridClock initialized: now 1790395675803155 us; error 47 us; skew 500 ppm
I20260926 04:07:55.804683 5433 webserver.cc:460] Webserver started at http://127.0.213.254:35537/ using document root <none> and password file <none>
I20260926 04:07:55.804970 5433 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:55.805042 5433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:55.805234 5433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:55.806385 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "03e978f5d3534c7a939fbfbfb2279043"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "8e69b0e1aea38ed8bc959da810d058c0"
server_key_iv: "2f23005838198c96cb6637389ac38983"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.806782 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "03e978f5d3534c7a939fbfbfb2279043"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "8e69b0e1aea38ed8bc959da810d058c0"
server_key_iv: "2f23005838198c96cb6637389ac38983"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.808238 5433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:55.809037 5447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.809253 5433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:55.809366 5433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "03e978f5d3534c7a939fbfbfb2279043"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "8e69b0e1aea38ed8bc959da810d058c0"
server_key_iv: "2f23005838198c96cb6637389ac38983"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.809446 5433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:55.812932 5433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:55.813220 5433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:55.813364 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:55.817402 5433 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:35455
I20260926 04:07:55.817436 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:35455 every 8 connection(s)
I20260926 04:07:55.817793 5433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:55.818574 5500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:55.821879 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043: Bootstrap starting.
I20260926 04:07:55.822643 5500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:55.822985 5500 log.cc:824] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:55.823777 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043: No bootstrap required, opened a new log
I20260926 04:07:55.825495 5500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e978f5d3534c7a939fbfbfb2279043" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35455 } }
I20260926 04:07:55.825641 5500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:55.825611 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5433
I20260926 04:07:55.825666 5500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e978f5d3534c7a939fbfbfb2279043, State: Initialized, Role: FOLLOWER
I20260926 04:07:55.825730 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:55.825917 855 external_mini_cluster.cc:1428] Setting key a4439acb8489a4f296bfb7823afa72ea
I20260926 04:07:55.825881 5500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e978f5d3534c7a939fbfbfb2279043" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35455 } }
I20260926 04:07:55.825968 5500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:55.826031 5500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:55.826078 5500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:55.826856 5500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e978f5d3534c7a939fbfbfb2279043" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35455 } }
I20260926 04:07:55.826965 5500 leader_election.cc:302] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03e978f5d3534c7a939fbfbfb2279043; no voters:
I20260926 04:07:55.827126 5500 leader_election.cc:288] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:55.827188 5505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:55.827378 5505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [term 1 LEADER]: Becoming Leader. State: Replica: 03e978f5d3534c7a939fbfbfb2279043, State: Running, Role: LEADER
I20260926 04:07:55.827512 5505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e978f5d3534c7a939fbfbfb2279043" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35455 } }
I20260926 04:07:55.827849 5500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:55.828105 5506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03e978f5d3534c7a939fbfbfb2279043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e978f5d3534c7a939fbfbfb2279043" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35455 } } }
I20260926 04:07:55.828200 5506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:55.828533 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.826435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.0.213.254:35455)
0926 04:07:55.826693 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.826704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.826742 (+ 38us) server_negotiation.cc:407] Connection header received
0926 04:07:55.826869 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.826877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.826925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.827079 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:55.827866 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:55.827870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:55.827870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:55.827876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:55.827923 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:55.827938 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:55.828034 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.828256 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.828323 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260926 04:07:55.828727 5507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03e978f5d3534c7a939fbfbfb2279043. Latest consensus state: current_term: 1 leader_uuid: "03e978f5d3534c7a939fbfbfb2279043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03e978f5d3534c7a939fbfbfb2279043" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35455 } } }
I20260926 04:07:55.828847 5507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:55.829032 5509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:55.829583 5509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:55.831398 5509 catalog_manager.cc:1347] Generated new cluster ID: 1dc8deadc27c46f3ad27edc6b64579ca
I20260926 04:07:55.831446 5509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:55.856781 5509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:55.856992 5509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:55.862687 5509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043: Generated new TSK 0
I20260926 04:07:55.862918 5509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:07:55.870493 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35455
--builtin_ntp_servers=127.0.213.212:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 04:07:55.953190 5524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:55.953441 5524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:55.953480 5524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:55.954679 5524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:55.954748 5524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:55.954798 5524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:55.956226 5524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:55.956501 5524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:55.956812 5524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:55.961947 5532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:55.961937 5529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.962049 5524 server_base.cc:1034] running on GCE node
W20260926 04:07:55.962139 5530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:55.962433 5524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:55.962683 5524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:55.963845 5524 hybrid_clock.cc:648] HybridClock initialized: now 1790395675963840 us; error 40 us; skew 500 ppm
I20260926 04:07:55.965227 5524 webserver.cc:460] Webserver started at http://127.0.213.193:38175/ using document root <none> and password file <none>
I20260926 04:07:55.965502 5524 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:55.965572 5524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:55.965723 5524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:55.966822 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "018bd658449247aa81dcba73fc4a7eac"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "b95bfbbc39f25c78db6297d62b0c4bab"
server_key_iv: "54878f0b1a1d1b74eca3e31c6d51d868"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.967192 5524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "018bd658449247aa81dcba73fc4a7eac"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "b95bfbbc39f25c78db6297d62b0c4bab"
server_key_iv: "54878f0b1a1d1b74eca3e31c6d51d868"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.968681 5524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:55.969522 5538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.969724 5524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:55.969842 5524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "018bd658449247aa81dcba73fc4a7eac"
format_stamp: "Formatted at 2026-09-26 04:07:55 on dist-test-slave-xf24"
server_key: "b95bfbbc39f25c78db6297d62b0c4bab"
server_key_iv: "54878f0b1a1d1b74eca3e31c6d51d868"
server_key_version: "encryptionkey@0"
I20260926 04:07:55.969916 5524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:55.973408 5524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:55.973702 5524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:55.973850 5524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:55.974067 5524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:55.974380 5524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:55.974433 5524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.974491 5524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:55.974520 5524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:55.980247 5524 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:43699
I20260926 04:07:55.980304 5651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:43699 every 8 connection(s)
I20260926 04:07:55.980661 5524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:55.983758 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.981415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:56771 (local address 127.0.213.254:35455)
0926 04:07:55.981482 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:55.981484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:55.981719 (+ 235us) server_negotiation.cc:407] Connection header received
0926 04:07:55.982217 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:55.982218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:55.982226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:55.982283 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:55.983500 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:55.983502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:55.983502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:55.983508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:55.983525 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:55.983530 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:55.983587 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:55.983669 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 04:07:55.983687 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:07:55.983982 5654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:55.981404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35455 (local address 127.0.213.193:56771)
0926 04:07:55.981600 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:55.981634 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:55.982113 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:55.982365 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:55.982368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:55.982377 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:55.983342 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:55.983343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:55.983373 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:55.983402 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:55.983581 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:55.983585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:55.983588 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:55.983680 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 04:07:55.983743 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260926 04:07:55.984544 5652 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35455
I20260926 04:07:55.984624 5652 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:55.984789 5652 heartbeater.cc:511] Master 127.0.213.254:35455 requested a full tablet report, sending...
I20260926 04:07:55.985302 5464 ts_manager.cc:194] Registered new tserver with Master: 018bd658449247aa81dcba73fc4a7eac (127.0.213.193:43699)
I20260926 04:07:55.985641 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5524
I20260926 04:07:55.985797 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:55.985986 855 external_mini_cluster.cc:1428] Setting key 9371d19613d87652f148bdfc01266181
I20260926 04:07:55.986881 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35455
--builtin_ntp_servers=127.0.213.212:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 04:07:56.067785 5655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:56.068004 5655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:56.068028 5655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:56.069263 5655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:56.069314 5655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:56.069339 5655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:56.070713 5655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:56.070990 5655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:56.071300 5655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:56.076485 5661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:56.076485 5660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:56.076576 5655 server_base.cc:1034] running on GCE node
W20260926 04:07:56.076534 5663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:56.076978 5655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:56.077235 5655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:56.078402 5655 hybrid_clock.cc:648] HybridClock initialized: now 1790395676078386 us; error 30 us; skew 500 ppm
I20260926 04:07:56.079771 5655 webserver.cc:460] Webserver started at http://127.0.213.194:46869/ using document root <none> and password file <none>
I20260926 04:07:56.080044 5655 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:56.080158 5655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:56.080302 5655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:56.081387 5655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "24fb2c9a464045f7bca97a41d6744340"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "ef838ace1ab3266e36e0825124721455"
server_key_iv: "3f29068ea90e9fbc28bc40fabdf9d6e1"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.081812 5655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "24fb2c9a464045f7bca97a41d6744340"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "ef838ace1ab3266e36e0825124721455"
server_key_iv: "3f29068ea90e9fbc28bc40fabdf9d6e1"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.083199 5655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:07:56.083976 5669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.084180 5655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:56.084280 5655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "24fb2c9a464045f7bca97a41d6744340"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "ef838ace1ab3266e36e0825124721455"
server_key_iv: "3f29068ea90e9fbc28bc40fabdf9d6e1"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.084374 5655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:56.087786 5655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:56.088131 5655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:56.088296 5655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:56.088510 5655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:56.088832 5655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:56.088887 5655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.088929 5655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:56.088958 5655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.094936 5655 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38907
I20260926 04:07:56.094985 5782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38907 every 8 connection(s)
I20260926 04:07:56.095353 5655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:56.099069 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.096288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:38891 (local address 127.0.213.254:35455)
0926 04:07:56.096408 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.096410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.096660 (+ 250us) server_negotiation.cc:407] Connection header received
0926 04:07:56.097175 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.097176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.097182 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.097254 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.098625 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.098627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.098627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.098638 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.098654 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.098658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.098722 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.098984 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.098999 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 04:07:56.099193 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.096304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35455 (local address 127.0.213.194:38891)
0926 04:07:56.096503 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.096546 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.097036 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.097362 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.097366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.097375 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.098417 (+ 1042us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.098417 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.098448 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.098482 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.098760 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.098766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.098770 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.098886 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.098947 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260926 04:07:56.100042 5783 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35455
I20260926 04:07:56.100235 5783 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:56.100445 5783 heartbeater.cc:511] Master 127.0.213.254:35455 requested a full tablet report, sending...
I20260926 04:07:56.100960 5464 ts_manager.cc:194] Registered new tserver with Master: 24fb2c9a464045f7bca97a41d6744340 (127.0.213.194:38907)
I20260926 04:07:56.101739 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5655
I20260926 04:07:56.101850 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:56.102036 855 external_mini_cluster.cc:1428] Setting key c5a9a0e430990c441ccaa87b0e583e7f
I20260926 04:07:56.103075 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35455
--builtin_ntp_servers=127.0.213.212:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 04:07:56.186879 5786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:56.187107 5786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:56.187131 5786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:56.188550 5786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:56.188599 5786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:56.188628 5786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:56.190049 5786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:56.190332 5786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:56.190653 5786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:07:56.195984 5786 server_base.cc:1034] running on GCE node
W20260926 04:07:56.195966 5791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:56.195966 5792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:56.195946 5794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:56.196494 5786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:56.196738 5786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:56.197880 5786 hybrid_clock.cc:648] HybridClock initialized: now 1790395676197869 us; error 32 us; skew 500 ppm
I20260926 04:07:56.199265 5786 webserver.cc:460] Webserver started at http://127.0.213.195:39661/ using document root <none> and password file <none>
I20260926 04:07:56.199522 5786 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:56.199570 5786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:56.199723 5786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:56.200933 5786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "d287fff26b624db49b0e5905609c2bee"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "e8ac45c136e595eca1d2b991a1e4acc6"
server_key_iv: "ec0018fe602c8d61cf232ed0f932f93d"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.201332 5786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "d287fff26b624db49b0e5905609c2bee"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "e8ac45c136e595eca1d2b991a1e4acc6"
server_key_iv: "ec0018fe602c8d61cf232ed0f932f93d"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.202782 5786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:07:56.203617 5800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.203810 5786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:56.203919 5786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "d287fff26b624db49b0e5905609c2bee"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "e8ac45c136e595eca1d2b991a1e4acc6"
server_key_iv: "ec0018fe602c8d61cf232ed0f932f93d"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.203999 5786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:56.207723 5786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:56.208092 5786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:56.208236 5786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:56.208468 5786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:56.208813 5786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:56.208868 5786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.208910 5786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:56.208935 5786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.215035 5786 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:41503
I20260926 04:07:56.215080 5913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:41503 every 8 connection(s)
I20260926 04:07:56.215426 5786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:07:56.218273 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5786
I20260926 04:07:56.218413 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:56.218611 855 external_mini_cluster.cc:1428] Setting key c2866feb1ccfbfc68bf893bb8bce86ec
I20260926 04:07:56.219089 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.216215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:38449 (local address 127.0.213.254:35455)
0926 04:07:56.216283 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.216286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.216638 (+ 352us) server_negotiation.cc:407] Connection header received
0926 04:07:56.217099 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.217100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.217105 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.217157 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.218717 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.218719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.218719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.218724 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.218742 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.218750 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.218850 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.218985 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.219004 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:07:56.219308 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.216248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35455 (local address 127.0.213.195:38449)
0926 04:07:56.216472 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.216519 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.217009 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.217216 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.217220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.217228 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.218484 (+ 1256us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.218485 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.218531 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.218583 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.218848 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.218852 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.218855 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.218958 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.219028 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260926 04:07:56.219784 5914 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35455
I20260926 04:07:56.219875 5914 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:56.220088 5914 heartbeater.cc:511] Master 127.0.213.254:35455 requested a full tablet report, sending...
I20260926 04:07:56.220515 5464 ts_manager.cc:194] Registered new tserver with Master: d287fff26b624db49b0e5905609c2bee (127.0.213.195:41503)
I20260926 04:07:56.230681 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:07:56.233284 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.232219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.0.213.254:35455)
0926 04:07:56.232261 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.232263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.232331 (+ 68us) server_negotiation.cc:407] Connection header received
0926 04:07:56.232373 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.232373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.232378 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.232426 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.233078 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.233079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.233079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.233082 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.233094 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.233098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.233138 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.233171 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.233184 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:07:56.234974 5464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:07:56.236579 5464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:07:56.240693 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.238888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38907 (local address 127.0.0.1:57678)
0926 04:07:56.239076 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.239120 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.239181 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.239539 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.239544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.239552 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.240223 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.240223 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.240242 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.240269 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.240491 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.240496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.240499 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.240580 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.240595 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260926 04:07:56.240903 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.238892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43699 (local address 127.0.0.1:44908)
0926 04:07:56.239265 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.239280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.239425 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.239910 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.239911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.239913 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.240515 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.240515 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.240521 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.240531 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.240777 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.240777 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.240778 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.240833 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.240846 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":91,"threads_started":1}
I20260926 04:07:56.240968 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.238971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.0.213.193:43699)
0926 04:07:56.239308 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.239312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.239365 (+ 53us) server_negotiation.cc:407] Connection header received
0926 04:07:56.239491 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.239498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.239509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.239594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.240615 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.240622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.240622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.240627 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.240664 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.240681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.240779 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.240842 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.240860 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":67,"threads_started":1}
I20260926 04:07:56.241020 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.239005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.213.194:38907)
0926 04:07:56.239404 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.239408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.239418 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:56.239444 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.239447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.239457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.239538 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.240334 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.240343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.240343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.240347 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.240389 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.240420 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.240474 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.240888 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.240904 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":155,"threads_started":1}
I20260926 04:07:56.241441 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.239112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:41503 (local address 127.0.0.1:47850)
0926 04:07:56.239962 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.239973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.240022 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.240565 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.240566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.240568 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.241041 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.241042 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.241048 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.241071 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.241317 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.241318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.241319 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.241371 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.241383 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":70,"threads_started":1}
I20260926 04:07:56.241715 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.239202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.213.195:41503)
0926 04:07:56.239393 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.239398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.240124 (+ 726us) server_negotiation.cc:407] Connection header received
0926 04:07:56.240157 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.240160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.240172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.240251 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.241159 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.241166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.241166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.241170 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.241210 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.241238 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.241294 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.241607 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.241624 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260926 04:07:56.241667 5717 tablet_service.cc:1467] Processing CreateTablet for tablet d049ab275f1143658ece75b96ce68eed (DEFAULT_TABLE table=test-table [id=48c0a344d7bb4b0bb5f5444467d5f089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:56.241657 5586 tablet_service.cc:1467] Processing CreateTablet for tablet d049ab275f1143658ece75b96ce68eed (DEFAULT_TABLE table=test-table [id=48c0a344d7bb4b0bb5f5444467d5f089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:56.242023 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d049ab275f1143658ece75b96ce68eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:56.242025 5717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d049ab275f1143658ece75b96ce68eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:56.244560 5928 tablet_bootstrap.cc:492] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Bootstrap starting.
I20260926 04:07:56.244565 5848 tablet_service.cc:1467] Processing CreateTablet for tablet d049ab275f1143658ece75b96ce68eed (DEFAULT_TABLE table=test-table [id=48c0a344d7bb4b0bb5f5444467d5f089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:07:56.244932 5848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d049ab275f1143658ece75b96ce68eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:56.244931 5929 tablet_bootstrap.cc:492] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: Bootstrap starting.
I20260926 04:07:56.245424 5928 tablet_bootstrap.cc:654] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:56.245563 5929 tablet_bootstrap.cc:654] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:56.245777 5928 log.cc:824] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:56.245885 5929 log.cc:824] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:56.246613 5929 tablet_bootstrap.cc:492] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: No bootstrap required, opened a new log
I20260926 04:07:56.246619 5928 tablet_bootstrap.cc:492] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: No bootstrap required, opened a new log
I20260926 04:07:56.246704 5929 ts_tablet_manager.cc:1397] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:56.246703 5928 ts_tablet_manager.cc:1397] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:07:56.248531 5932 tablet_bootstrap.cc:492] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Bootstrap starting.
I20260926 04:07:56.248857 5929 raft_consensus.cc:348] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.249161 5929 raft_consensus.cc:374] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:56.249230 5929 raft_consensus.cc:729] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 018bd658449247aa81dcba73fc4a7eac, State: Initialized, Role: FOLLOWER
I20260926 04:07:56.249289 5932 tablet_bootstrap.cc:654] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:56.249387 5929 consensus_queue.cc:260] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.249485 5928 raft_consensus.cc:348] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.249667 5932 log.cc:824] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:56.249706 5929 ts_tablet_manager.cc:1428] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:56.249724 5652 heartbeater.cc:503] Master 127.0.213.254:35455 was elected leader, sending a full tablet report...
I20260926 04:07:56.249979 5928 raft_consensus.cc:374] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:56.250072 5928 raft_consensus.cc:729] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fb2c9a464045f7bca97a41d6744340, State: Initialized, Role: FOLLOWER
I20260926 04:07:56.250222 5928 consensus_queue.cc:260] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.250499 5928 ts_tablet_manager.cc:1428] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 04:07:56.250607 5783 heartbeater.cc:503] Master 127.0.213.254:35455 was elected leader, sending a full tablet report...
I20260926 04:07:56.250891 5932 tablet_bootstrap.cc:492] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: No bootstrap required, opened a new log
I20260926 04:07:56.250972 5932 ts_tablet_manager.cc:1397] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:07:56.252951 5932 raft_consensus.cc:348] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.253113 5932 raft_consensus.cc:374] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:56.253161 5932 raft_consensus.cc:729] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d287fff26b624db49b0e5905609c2bee, State: Initialized, Role: FOLLOWER
I20260926 04:07:56.253276 5932 consensus_queue.cc:260] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.253545 5932 ts_tablet_manager.cc:1428] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:07:56.253611 5914 heartbeater.cc:503] Master 127.0.213.254:35455 was elected leader, sending a full tablet report...
W20260926 04:07:56.345968 5784 tablet.cc:2369] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:07:56.399077 5935 raft_consensus.cc:482] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:07:56.399212 5935 raft_consensus.cc:504] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.399544 5935 leader_election.cc:288] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 018bd658449247aa81dcba73fc4a7eac (127.0.213.193:43699), d287fff26b624db49b0e5905609c2bee (127.0.213.195:41503)
I20260926 04:07:56.401180 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.399756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:41503 (local address 127.0.213.194:38793)
0926 04:07:56.399796 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.399809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.399850 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.400134 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.400134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.400153 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.400782 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.400782 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.400787 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.400797 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.401011 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.401012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.401013 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.401068 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.401086 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 04:07:56.401187 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.399849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:48167 (local address 127.0.213.193:43699)
0926 04:07:56.399893 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.399896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.400043 (+ 147us) server_negotiation.cc:407] Connection header received
0926 04:07:56.400094 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.400096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.400102 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.400166 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.400885 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.400887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.400887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.400892 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.400920 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.400925 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.400996 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.401091 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.401109 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:07:56.401194 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.399843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:38793 (local address 127.0.213.195:41503)
0926 04:07:56.399900 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.399902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.399911 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:07:56.399951 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.399952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.399958 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.400127 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.400894 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.400895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.400896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.400900 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.400920 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.400925 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.400996 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.401093 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.401114 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:07:56.401233 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.399756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43699 (local address 127.0.213.194:48167)
0926 04:07:56.399958 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.399965 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.400001 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.400181 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.400181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.400183 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:56.400726 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:07:56.400727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:56.400750 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:07:56.400767 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:56.401011 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:56.401012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:56.401013 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 04:07:56.401070 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:07:56.401087 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":50,"thread_start_us":91,"threads_started":1}
I20260926 04:07:56.401569 5868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d049ab275f1143658ece75b96ce68eed" candidate_uuid: "24fb2c9a464045f7bca97a41d6744340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d287fff26b624db49b0e5905609c2bee" is_pre_election: true
I20260926 04:07:56.401566 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d049ab275f1143658ece75b96ce68eed" candidate_uuid: "24fb2c9a464045f7bca97a41d6744340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "018bd658449247aa81dcba73fc4a7eac" is_pre_election: true
I20260926 04:07:56.401731 5606 raft_consensus.cc:2454] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fb2c9a464045f7bca97a41d6744340 in term 0.
I20260926 04:07:56.401732 5868 raft_consensus.cc:2454] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fb2c9a464045f7bca97a41d6744340 in term 0.
I20260926 04:07:56.401993 5671 leader_election.cc:302] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 018bd658449247aa81dcba73fc4a7eac, 24fb2c9a464045f7bca97a41d6744340; no voters:
I20260926 04:07:56.402124 5935 raft_consensus.cc:2781] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:07:56.402212 5935 raft_consensus.cc:482] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:07:56.402251 5935 raft_consensus.cc:3037] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:56.403052 5935 raft_consensus.cc:504] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.403316 5935 leader_election.cc:288] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [CANDIDATE]: Term 1 election: Requested vote from peers 018bd658449247aa81dcba73fc4a7eac (127.0.213.193:43699), d287fff26b624db49b0e5905609c2bee (127.0.213.195:41503)
I20260926 04:07:56.403434 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d049ab275f1143658ece75b96ce68eed" candidate_uuid: "24fb2c9a464045f7bca97a41d6744340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "018bd658449247aa81dcba73fc4a7eac"
I20260926 04:07:56.403482 5868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d049ab275f1143658ece75b96ce68eed" candidate_uuid: "24fb2c9a464045f7bca97a41d6744340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d287fff26b624db49b0e5905609c2bee"
I20260926 04:07:56.403555 5606 raft_consensus.cc:3037] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:56.403558 5868 raft_consensus.cc:3037] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:56.404371 5606 raft_consensus.cc:2454] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fb2c9a464045f7bca97a41d6744340 in term 1.
I20260926 04:07:56.404386 5868 raft_consensus.cc:2454] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fb2c9a464045f7bca97a41d6744340 in term 1.
I20260926 04:07:56.404579 5672 leader_election.cc:302] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fb2c9a464045f7bca97a41d6744340, d287fff26b624db49b0e5905609c2bee; no voters:
I20260926 04:07:56.404680 5935 raft_consensus.cc:2781] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:56.404842 5935 raft_consensus.cc:686] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 1 LEADER]: Becoming Leader. State: Replica: 24fb2c9a464045f7bca97a41d6744340, State: Running, Role: LEADER
I20260926 04:07:56.405062 5935 consensus_queue.cc:237] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } }
I20260926 04:07:56.406119 5463 catalog_manager.cc:5495] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fb2c9a464045f7bca97a41d6744340 (127.0.213.194). New cstate: current_term: 1 leader_uuid: "24fb2c9a464045f7bca97a41d6744340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fb2c9a464045f7bca97a41d6744340" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260926 04:07:56.431638 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.430456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.0.213.194:38907)
0926 04:07:56.430557 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.430559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.430567 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:07:56.430612 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.430613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.430626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.430694 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.431382 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.431384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.431384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.431389 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.431406 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.431410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.431489 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.431538 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.431555 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 04:07:56.433710 5868 raft_consensus.cc:1261] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 1 FOLLOWER]: Refusing update from remote peer 24fb2c9a464045f7bca97a41d6744340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:56.433710 5606 raft_consensus.cc:1261] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac [term 1 FOLLOWER]: Refusing update from remote peer 24fb2c9a464045f7bca97a41d6744340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:07:56.434113 5938 consensus_queue.cc:1035] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d287fff26b624db49b0e5905609c2bee" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:56.434257 5935 consensus_queue.cc:1035] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "018bd658449247aa81dcba73fc4a7eac" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:07:56.436841 5941 mvcc.cc:205] Tried to move back new op lower bound from 7333460690669293568 to 7333460690556260352. Current Snapshot: MvccSnapshot[applied={T|T < 7333460690669293568}]
I20260926 04:07:56.440150 5463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:56.440267 5463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:07:56.441098 5463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03e978f5d3534c7a939fbfbfb2279043: Sending DeleteTablet for 3 replicas of tablet d049ab275f1143658ece75b96ce68eed
I20260926 04:07:56.441473 5717 tablet_service.cc:1514] Processing DeleteTablet for tablet d049ab275f1143658ece75b96ce68eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:56 UTC) from {username='slave'} at 127.0.0.1:57678
I20260926 04:07:56.441489 5586 tablet_service.cc:1514] Processing DeleteTablet for tablet d049ab275f1143658ece75b96ce68eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:56 UTC) from {username='slave'} at 127.0.0.1:44908
I20260926 04:07:56.441541 5848 tablet_service.cc:1514] Processing DeleteTablet for tablet d049ab275f1143658ece75b96ce68eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:07:56 UTC) from {username='slave'} at 127.0.0.1:47850
I20260926 04:07:56.441777 5952 tablet_replica.cc:330] T d049ab275f1143658ece75b96ce68eed P 018bd658449247aa81dcba73fc4a7eac: stopping tablet replica
I20260926 04:07:56.441836 5953 tablet_replica.cc:330] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: stopping tablet replica
I20260926 04:07:56.441922 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5524
I20260926 04:07:56.442559 5951 tablet_replica.cc:330] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: stopping tablet replica
I20260926 04:07:56.442739 5951 raft_consensus.cc:2229] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:07:56.442888 5953 raft_consensus.cc:2229] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:07:56.442955 5951 raft_consensus.cc:2258] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:56.443040 5953 raft_consensus.cc:2258] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:07:56.443552 5951 ts_tablet_manager.cc:1905] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:56.445164 5953 ts_tablet_manager.cc:1905] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:07:56.445371 5951 ts_tablet_manager.cc:1918] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:56.445446 5951 log.cc:1196] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/d049ab275f1143658ece75b96ce68eed
I20260926 04:07:56.445748 5951 ts_tablet_manager.cc:1939] T d049ab275f1143658ece75b96ce68eed P 24fb2c9a464045f7bca97a41d6744340: Deleting consensus metadata
I20260926 04:07:56.446195 5450 catalog_manager.cc:4841] TS 24fb2c9a464045f7bca97a41d6744340 (127.0.213.194:38907): tablet d049ab275f1143658ece75b96ce68eed (table test-table [id=48c0a344d7bb4b0bb5f5444467d5f089]) successfully deleted
W20260926 04:07:56.446713 5449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:43699 (error 108)
I20260926 04:07:56.446796 5953 ts_tablet_manager.cc:1918] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:07:56.446868 5953 log.cc:1196] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/d049ab275f1143658ece75b96ce68eed
I20260926 04:07:56.447121 5953 ts_tablet_manager.cc:1939] T d049ab275f1143658ece75b96ce68eed P d287fff26b624db49b0e5905609c2bee: Deleting consensus metadata
I20260926 04:07:56.447606 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.447457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43699 (local address 127.0.0.1:44920)
0926 04:07:56.447522 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.447571 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 04:07:56.447705 5450 catalog_manager.cc:4841] TS d287fff26b624db49b0e5905609c2bee (127.0.213.195:41503): tablet d049ab275f1143658ece75b96ce68eed (table test-table [id=48c0a344d7bb4b0bb5f5444467d5f089]) successfully deleted
W20260926 04:07:56.447729 5449 catalog_manager.cc:4584] TS 018bd658449247aa81dcba73fc4a7eac (127.0.213.193:43699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d049ab275f1143658ece75b96ce68eed: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43699: connect: Connection refused (error 111)
I20260926 04:07:56.448112 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5655
I20260926 04:07:56.453455 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5786
I20260926 04:07:56.458087 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5433
2026-09-26T04:07:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (775 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 04:07:56.477842 855 mini_oidc.cc:152] Starting JWKS server
I20260926 04:07:56.478941 855 webserver.cc:460] Webserver started at https://127.0.0.1:40145/ using document root <none> and password file <none>
I20260926 04:07:56.479077 855 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 04:07:56.479285 855 webserver.cc:460] Webserver started at http://127.0.0.1:37443/ using document root <none> and password file <none>
2026-09-26T04:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:56Z Disabled control of system clock
I20260926 04:07:56.487393 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:44501
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:44501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40145/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/testchainca.cert with env {}
W20260926 04:07:56.568228 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:56.568506 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:56.568544 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:56.569790 5960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:56.569868 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:56.569890 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:56.569909 5960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:56.569938 5960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 04:07:56.569967 5960 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 04:07:56.569990 5960 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40145/jwks.json
I20260926 04:07:56.571604 5960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:44501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:44501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40145/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:56.571911 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:56.572167 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:56.577150 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:56.577211 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:56.577150 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:56.577431 5960 server_base.cc:1034] running on GCE node
I20260926 04:07:56.577728 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:56.578040 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:56.579185 5960 hybrid_clock.cc:648] HybridClock initialized: now 1790395676579170 us; error 31 us; skew 500 ppm
I20260926 04:07:56.580607 5960 webserver.cc:460] Webserver started at http://127.0.213.254:44303/ using document root <none> and password file <none>
I20260926 04:07:56.580830 5960 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:56.580874 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:56.580966 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:56.582020 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "c6771c148e8344aa8bb95534074a8577"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "f52f441bbaa22a8bab37245340396615"
server_key_iv: "b4ca1030a9135e04d47fc4594a9fd79f"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.582326 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "c6771c148e8344aa8bb95534074a8577"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "f52f441bbaa22a8bab37245340396615"
server_key_iv: "b4ca1030a9135e04d47fc4594a9fd79f"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.583658 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:56.584523 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.584733 5960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:07:56.584834 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "c6771c148e8344aa8bb95534074a8577"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "f52f441bbaa22a8bab37245340396615"
server_key_iv: "b4ca1030a9135e04d47fc4594a9fd79f"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.584900 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:56.617694 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:56.618005 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:56.618175 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:56.622157 5960 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:44501
I20260926 04:07:56.622224 6026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:44501 every 8 connection(s)
I20260926 04:07:56.622560 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:56.623567 6027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:56.625995 6027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577: Bootstrap starting.
I20260926 04:07:56.626618 6027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:56.626926 6027 log.cc:824] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:56.627641 6027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577: No bootstrap required, opened a new log
I20260926 04:07:56.629374 6027 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6771c148e8344aa8bb95534074a8577" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44501 } }
I20260926 04:07:56.629544 6027 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:56.629583 6027 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6771c148e8344aa8bb95534074a8577, State: Initialized, Role: FOLLOWER
I20260926 04:07:56.629769 6027 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6771c148e8344aa8bb95534074a8577" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44501 } }
I20260926 04:07:56.629868 6027 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:56.629913 6027 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:56.629962 6027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:56.630494 6027 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6771c148e8344aa8bb95534074a8577" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44501 } }
I20260926 04:07:56.630615 6027 leader_election.cc:302] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6771c148e8344aa8bb95534074a8577; no voters:
I20260926 04:07:56.630820 6027 leader_election.cc:288] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:56.630878 6030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:56.631078 6027 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:56.631110 6030 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [term 1 LEADER]: Becoming Leader. State: Replica: c6771c148e8344aa8bb95534074a8577, State: Running, Role: LEADER
I20260926 04:07:56.631304 6030 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6771c148e8344aa8bb95534074a8577" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44501 } }
I20260926 04:07:56.631775 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6771c148e8344aa8bb95534074a8577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6771c148e8344aa8bb95534074a8577" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44501 } } }
I20260926 04:07:56.631870 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:56.632349 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 5960
I20260926 04:07:56.631870 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6771c148e8344aa8bb95534074a8577. Latest consensus state: current_term: 1 leader_uuid: "c6771c148e8344aa8bb95534074a8577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6771c148e8344aa8bb95534074a8577" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44501 } } }
I20260926 04:07:56.632437 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:56.632459 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:56.632652 855 external_mini_cluster.cc:1428] Setting key df056e31908800a1811d0e796a134c3f
W20260926 04:07:56.633121 6046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:07:56.633235 6046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:07:56.633316 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:56.633560 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:56.635918 6039 catalog_manager.cc:1347] Generated new cluster ID: cc36c35c55d5471e9b830d4bc9fce5b1
I20260926 04:07:56.635987 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:56.639043 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.633179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.0.213.254:44501)
0926 04:07:56.633409 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.633419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.633450 (+ 31us) server_negotiation.cc:407] Connection header received
0926 04:07:56.633570 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.633577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.633649 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.633798 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.634059 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.635389 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.636734 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.637403 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.637544 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:56.638296 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.638302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.638302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.638520 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.638566 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.638582 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.638653 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.638731 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.638824 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260926 04:07:56.640219 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:07:56.646462 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:56.647202 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:56.653532 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6771c148e8344aa8bb95534074a8577: Generated new TSK 0
I20260926 04:07:56.653795 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 04:07:56.679651 6057 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 04:07:56.679831 6057 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 04:07:56.680089 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.664083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.0.213.254:44501)
0926 04:07:56.664140 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.664142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.664261 (+ 119us) server_negotiation.cc:407] Connection header received
0926 04:07:56.664332 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.664333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.664346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.664427 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0926 04:07:56.664583 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.665369 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.666195 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.666506 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.666580 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:56.666818 (+ 238us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 04:07:56.679804 (+ 12986us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 04:07:56.679962 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 04:07:56.680156 6049 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 04:07:56.680573 6056 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:56.663970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:44501 (local address 127.0.0.1:35284)
0926 04:07:56.664174 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.664189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.664251 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.664440 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.664441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.664448 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 04:07:56.664508 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:56.664511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.665466 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.665468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:56.666094 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:56.666096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.666603 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.666604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:56.666665 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:56.666687 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 04:07:56.680038 (+ 13351us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 04:07:56.680243 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.213.254:44501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":131,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.213.254:44501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 04:07:56.681077 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 5960
2026-09-26T04:07:56Z chronyd exiting
I20260926 04:07:56.717387 855 test_util.cc:175] -----------------------------------------------
I20260926 04:07:56.717471 855 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395654118158-855-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 04:07:56.718254 855 mini_oidc.cc:152] Starting JWKS server
I20260926 04:07:56.718989 855 webserver.cc:460] Webserver started at https://127.0.0.1:42545/ using document root <none> and password file <none>
I20260926 04:07:56.719062 855 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 04:07:56.719259 855 webserver.cc:460] Webserver started at http://127.0.0.1:45077/ using document root <none> and password file <none>
2026-09-26T04:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:56Z Disabled control of system clock
I20260926 04:07:56.726912 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40747
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/testchainca.cert with env {}
W20260926 04:07:56.808770 6063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:56.809038 6063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:56.809082 6063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:56.810389 6063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:56.810464 6063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:56.810487 6063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:56.810509 6063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:56.810539 6063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 04:07:56.810565 6063 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 04:07:56.810590 6063 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42545/jwks.json
I20260926 04:07:56.813227 6063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:56.813544 6063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:56.813792 6063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:56.819027 6071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:56.819022 6069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:56.819146 6063 server_base.cc:1034] running on GCE node
W20260926 04:07:56.819022 6068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:56.819653 6063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:56.819942 6063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:56.821165 6063 hybrid_clock.cc:648] HybridClock initialized: now 1790395676821151 us; error 32 us; skew 500 ppm
I20260926 04:07:56.822603 6063 webserver.cc:460] Webserver started at http://127.0.213.254:33317/ using document root <none> and password file <none>
I20260926 04:07:56.822860 6063 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:56.822932 6063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:56.823099 6063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:56.824324 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "d2fa69487e804fdd8f18bd5ab96777eb"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "c37b02cf5318116976877d2f6bc0656a"
server_key_iv: "4206d6d3d5c58b4424901b8c429b2bab"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.824723 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "d2fa69487e804fdd8f18bd5ab96777eb"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "c37b02cf5318116976877d2f6bc0656a"
server_key_iv: "4206d6d3d5c58b4424901b8c429b2bab"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.826160 6063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:07:56.826939 6077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:56.827126 6063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:56.827236 6063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "d2fa69487e804fdd8f18bd5ab96777eb"
format_stamp: "Formatted at 2026-09-26 04:07:56 on dist-test-slave-xf24"
server_key: "c37b02cf5318116976877d2f6bc0656a"
server_key_iv: "4206d6d3d5c58b4424901b8c429b2bab"
server_key_version: "encryptionkey@0"
I20260926 04:07:56.827315 6063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:56.839637 6063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:56.840022 6063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:56.840287 6063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:56.844722 6063 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:40747
I20260926 04:07:56.844749 6129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:40747 every 8 connection(s)
I20260926 04:07:56.845111 6063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:56.846033 6130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:56.848493 6130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb: Bootstrap starting.
I20260926 04:07:56.849102 6130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:56.849413 6130 log.cc:824] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:56.850244 6130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb: No bootstrap required, opened a new log
I20260926 04:07:56.851622 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6063
I20260926 04:07:56.851709 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:56.851912 855 external_mini_cluster.cc:1428] Setting key e95128e579323b435cad570541ea4f40
I20260926 04:07:56.852166 6130 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40747 } }
I20260926 04:07:56.852337 6130 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:56.852676 6130 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fa69487e804fdd8f18bd5ab96777eb, State: Initialized, Role: FOLLOWER
I20260926 04:07:56.852807 6130 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40747 } }
I20260926 04:07:56.852893 6130 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:56.852931 6130 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:56.852978 6130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:56.853720 6130 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40747 } }
I20260926 04:07:56.853808 6130 leader_election.cc:302] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fa69487e804fdd8f18bd5ab96777eb; no voters:
I20260926 04:07:56.854017 6130 leader_election.cc:288] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:56.854048 6135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:56.854151 6135 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [term 1 LEADER]: Becoming Leader. State: Replica: d2fa69487e804fdd8f18bd5ab96777eb, State: Running, Role: LEADER
I20260926 04:07:56.854295 6130 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:56.854302 6135 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40747 } }
I20260926 04:07:56.854678 6135 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fa69487e804fdd8f18bd5ab96777eb. Latest consensus state: current_term: 1 leader_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40747 } } }
I20260926 04:07:56.854707 6136 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fa69487e804fdd8f18bd5ab96777eb" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40747 } } }
I20260926 04:07:56.854821 6135 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:56.854831 6136 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:56.855114 6143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:56.856254 6143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:56.858174 6143 catalog_manager.cc:1347] Generated new cluster ID: 5ba4060b06934c4295127cfe645c9339
I20260926 04:07:56.858240 6143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:56.858912 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.852464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.0.213.254:40747)
0926 04:07:56.852675 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.852685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.852726 (+ 41us) server_negotiation.cc:407] Connection header received
0926 04:07:56.852846 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.852854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.852930 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.853093 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:56.853465 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.855111 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.856333 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.857044 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.857181 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:56.857972 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:56.857976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:56.857977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:56.858201 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:56.858245 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:56.858261 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:56.858331 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:56.858617 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 04:07:56.858703 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260926 04:07:56.859920 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:07:56.872980 6143 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:56.873581 6143 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:56.876488 6143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2fa69487e804fdd8f18bd5ab96777eb: Generated new TSK 0
I20260926 04:07:56.876696 6143 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 04:07:56.900256 6160 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 04:07:56.900350 6160 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 04:07:56.900707 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:56.886196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.0.213.254:40747)
0926 04:07:56.886266 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:56.886268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:56.886395 (+ 127us) server_negotiation.cc:407] Connection header received
0926 04:07:56.886490 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:56.886491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:56.886503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:56.886571 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0926 04:07:56.886721 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.887222 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.888413 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.888628 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.888694 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:56.888878 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 04:07:56.900518 (+ 11640us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 04:07:56.900654 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 04:07:56.900779 6134 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 04:07:56.900889 6159 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:56.886109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40747 (local address 127.0.0.1:54076)
0926 04:07:56.886303 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:56.886315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:56.886390 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:56.886589 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:56.886590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:56.886592 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 04:07:56.886652 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:56.886655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.887306 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.887307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:56.888319 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:56.888322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:56.888711 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:56.888712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:56.888774 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:56.888780 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 04:07:56.900642 (+ 11862us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 04:07:56.900782 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.213.254:40747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":116,"threads_started":1}
I20260926 04:07:56.901196 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6063
2026-09-26T04:07:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 04:07:56.938762 855 mini_oidc.cc:152] Starting JWKS server
I20260926 04:07:56.939479 855 webserver.cc:460] Webserver started at https://127.0.0.1:44631/ using document root <none> and password file <none>
I20260926 04:07:56.939529 855 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 04:07:56.939728 855 webserver.cc:460] Webserver started at http://127.0.0.1:35269/ using document root <none> and password file <none>
2026-09-26T04:07:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:56Z Disabled control of system clock
I20260926 04:07:56.947925 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:38017
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:39809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:38017
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44631/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 04:07:57.029194 6166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:57.029464 6166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:57.029502 6166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:57.030752 6166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:57.030835 6166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:57.030856 6166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:57.030879 6166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:57.030905 6166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 04:07:57.030936 6166 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 04:07:57.030965 6166 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44631/jwks.json
I20260926 04:07:57.032445 6166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:38017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44631/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:57.032749 6166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:57.033062 6166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:57.038403 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:57.038389 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:57.038419 6166 server_base.cc:1034] running on GCE node
W20260926 04:07:57.038388 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:57.038911 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:57.039222 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:57.040410 6166 hybrid_clock.cc:648] HybridClock initialized: now 1790395677040388 us; error 51 us; skew 500 ppm
I20260926 04:07:57.041848 6166 webserver.cc:460] Webserver started at http://127.0.213.254:42359/ using document root <none> and password file <none>
I20260926 04:07:57.042119 6166 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:57.042191 6166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:57.042346 6166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:57.043455 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "6e63caa00d4d4165aaed0f5bec0d8849"
format_stamp: "Formatted at 2026-09-26 04:07:57 on dist-test-slave-xf24"
server_key: "e50932c3d598625af232b9c4386a9e9c"
server_key_iv: "30311d5a67cfb47ce2c1ed7418882a56"
server_key_version: "encryptionkey@0"
I20260926 04:07:57.043833 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "6e63caa00d4d4165aaed0f5bec0d8849"
format_stamp: "Formatted at 2026-09-26 04:07:57 on dist-test-slave-xf24"
server_key: "e50932c3d598625af232b9c4386a9e9c"
server_key_iv: "30311d5a67cfb47ce2c1ed7418882a56"
server_key_version: "encryptionkey@0"
I20260926 04:07:57.045325 6166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:57.046142 6180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:57.046432 6166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:57.046545 6166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "6e63caa00d4d4165aaed0f5bec0d8849"
format_stamp: "Formatted at 2026-09-26 04:07:57 on dist-test-slave-xf24"
server_key: "e50932c3d598625af232b9c4386a9e9c"
server_key_iv: "30311d5a67cfb47ce2c1ed7418882a56"
server_key_version: "encryptionkey@0"
I20260926 04:07:57.046659 6166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:57.068612 6166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:57.068948 6166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:57.069065 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:57.072922 6166 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:38017
I20260926 04:07:57.072955 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:38017 every 8 connection(s)
I20260926 04:07:57.073323 6166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:57.074026 6233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:57.076798 6233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849: Bootstrap starting.
I20260926 04:07:57.077396 6233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:57.077690 6233 log.cc:824] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:57.078409 6233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849: No bootstrap required, opened a new log
I20260926 04:07:57.080199 6233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38017 } }
I20260926 04:07:57.080343 6233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:57.080365 6233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e63caa00d4d4165aaed0f5bec0d8849, State: Initialized, Role: FOLLOWER
I20260926 04:07:57.080535 6233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38017 } }
I20260926 04:07:57.080622 6233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:57.080657 6233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:57.080727 6233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:57.081284 6233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38017 } }
I20260926 04:07:57.081398 6233 leader_election.cc:302] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e63caa00d4d4165aaed0f5bec0d8849; no voters:
I20260926 04:07:57.081586 6233 leader_election.cc:288] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:57.081697 6236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:57.081845 6233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:57.081877 6236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [term 1 LEADER]: Becoming Leader. State: Replica: 6e63caa00d4d4165aaed0f5bec0d8849, State: Running, Role: LEADER
I20260926 04:07:57.081979 6236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38017 } }
I20260926 04:07:57.082549 6237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38017 } } }
I20260926 04:07:57.082640 6237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:57.082891 6245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:57.082976 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6166
I20260926 04:07:57.083050 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:57.083225 855 external_mini_cluster.cc:1428] Setting key cf2318e9ffb24870d81893ee1240b4b6
I20260926 04:07:57.083439 6245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:57.082762 6238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e63caa00d4d4165aaed0f5bec0d8849. Latest consensus state: current_term: 1 leader_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e63caa00d4d4165aaed0f5bec0d8849" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38017 } } }
I20260926 04:07:57.083576 6238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849 [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:57.085152 6245 catalog_manager.cc:1347] Generated new cluster ID: 97084f9bfe6f430b97b7868827aeafd2
I20260926 04:07:57.085211 6245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:57.089543 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:57.083680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.213.254:38017)
0926 04:07:57.084190 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:57.084199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:57.084230 (+ 31us) server_negotiation.cc:407] Connection header received
0926 04:07:57.084353 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:57.084360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:57.084431 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:57.084575 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:57.084818 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.086191 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.087276 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.087925 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.088108 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:57.088796 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:57.088800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:57.088800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:57.089026 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:57.089069 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:57.089083 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:57.089156 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:57.089232 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 04:07:57.089313 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":69,"threads_started":1}
I20260926 04:07:57.090487 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:07:57.093006 6245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:57.093482 6245 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:57.100268 6245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e63caa00d4d4165aaed0f5bec0d8849: Generated new TSK 0
I20260926 04:07:57.100498 6245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:07:57.109642 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:57.105726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.213.254:38017)
0926 04:07:57.105793 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:57.105795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:57.105944 (+ 149us) server_negotiation.cc:407] Connection header received
0926 04:07:57.106026 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:57.106027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:57.106038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:57.106111 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0926 04:07:57.106256 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.106745 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.107918 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.108187 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.108252 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:57.108480 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 04:07:57.109373 (+ 893us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 04:07:57.109531 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47692: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 04:07:57.109685 6262 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:57.105670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:38017 (local address 127.0.0.1:47692)
0926 04:07:57.105862 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:57.105874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:57.105945 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:57.106122 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:57.106123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:57.106126 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 04:07:57.106191 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:57.106194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.106844 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.106845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:57.107820 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:57.107824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.108275 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.108277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:57.108343 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:57.108350 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 04:07:57.109485 (+ 1135us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 04:07:57.109586 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.213.254:38017: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260926 04:07:57.109863 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6166
E20260926 04:07:57.122323 6263 webserver.cc:533] Webserver: sslize failed:
E20260926 04:07:57.122413 6263 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T04:07:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T04:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:57Z Disabled control of system clock
I20260926 04:07:57.156118 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36453
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:36453
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/testchainca.cert with env {}
W20260926 04:07:57.239609 6267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:57.239836 6267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:57.239861 6267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:57.241182 6267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:57.241236 6267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:57.241250 6267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:57.241262 6267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:57.241276 6267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 04:07:57.241287 6267 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 04:07:57.241299 6267 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 04:07:57.242789 6267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:36453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:57.243009 6267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:57.243186 6267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:57.248605 6275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:57.248692 6272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:57.248692 6273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:57.248869 6267 server_base.cc:1034] running on GCE node
I20260926 04:07:57.249145 6267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:57.249485 6267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:57.250705 6267 hybrid_clock.cc:648] HybridClock initialized: now 1790395677250687 us; error 29 us; skew 500 ppm
I20260926 04:07:57.252382 6267 webserver.cc:460] Webserver started at http://127.0.213.254:40885/ using document root <none> and password file <none>
I20260926 04:07:57.252667 6267 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:57.252748 6267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:57.252921 6267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:57.254150 6267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "63d3779f843c47f7a6eb54375e59e1cd"
format_stamp: "Formatted at 2026-09-26 04:07:57 on dist-test-slave-xf24"
server_key: "5724d27d949ecfc409116f5fb4c2e31f"
server_key_iv: "e0e9c1a830b3760c429e5abd2c803b0c"
server_key_version: "encryptionkey@0"
I20260926 04:07:57.254565 6267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "63d3779f843c47f7a6eb54375e59e1cd"
format_stamp: "Formatted at 2026-09-26 04:07:57 on dist-test-slave-xf24"
server_key: "5724d27d949ecfc409116f5fb4c2e31f"
server_key_iv: "e0e9c1a830b3760c429e5abd2c803b0c"
server_key_version: "encryptionkey@0"
I20260926 04:07:57.256225 6267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 04:07:57.257130 6281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:57.257337 6267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:57.257449 6267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "63d3779f843c47f7a6eb54375e59e1cd"
format_stamp: "Formatted at 2026-09-26 04:07:57 on dist-test-slave-xf24"
server_key: "5724d27d949ecfc409116f5fb4c2e31f"
server_key_iv: "e0e9c1a830b3760c429e5abd2c803b0c"
server_key_version: "encryptionkey@0"
I20260926 04:07:57.257529 6267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:57.267509 6267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:57.267877 6267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:57.268095 6267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:57.272740 6267 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:36453
I20260926 04:07:57.272840 6333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:36453 every 8 connection(s)
I20260926 04:07:57.273238 6267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:57.274377 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:57.276952 6334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd: Bootstrap starting.
I20260926 04:07:57.277596 6334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:57.277900 6334 log.cc:824] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:57.278753 6334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd: No bootstrap required, opened a new log
I20260926 04:07:57.280912 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6267
I20260926 04:07:57.281018 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:57.280829 6334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d3779f843c47f7a6eb54375e59e1cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36453 } }
I20260926 04:07:57.281096 6334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:57.281138 6334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d3779f843c47f7a6eb54375e59e1cd, State: Initialized, Role: FOLLOWER
I20260926 04:07:57.281207 855 external_mini_cluster.cc:1428] Setting key 7d0ef857beb4e5ee233b45759ee8c935
I20260926 04:07:57.281324 6334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d3779f843c47f7a6eb54375e59e1cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36453 } }
I20260926 04:07:57.281414 6334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:57.281464 6334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:57.281517 6334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:57.282325 6334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d3779f843c47f7a6eb54375e59e1cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36453 } }
I20260926 04:07:57.282466 6334 leader_election.cc:302] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d3779f843c47f7a6eb54375e59e1cd; no voters:
I20260926 04:07:57.282666 6334 leader_election.cc:288] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:57.282718 6339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:57.282871 6339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [term 1 LEADER]: Becoming Leader. State: Replica: 63d3779f843c47f7a6eb54375e59e1cd, State: Running, Role: LEADER
I20260926 04:07:57.282965 6339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d3779f843c47f7a6eb54375e59e1cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36453 } }
I20260926 04:07:57.282964 6334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:57.283352 6341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d3779f843c47f7a6eb54375e59e1cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d3779f843c47f7a6eb54375e59e1cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36453 } } }
I20260926 04:07:57.283352 6340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d3779f843c47f7a6eb54375e59e1cd. Latest consensus state: current_term: 1 leader_uuid: "63d3779f843c47f7a6eb54375e59e1cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d3779f843c47f7a6eb54375e59e1cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36453 } } }
I20260926 04:07:57.283509 6340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:57.283483 6341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:57.283725 6346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:57.284494 6346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:57.286204 6346 catalog_manager.cc:1347] Generated new cluster ID: c502774671c74cadadace8a24f0ad7bb
I20260926 04:07:57.286572 6346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:07:57.287590 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:57.281710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.0.213.254:36453)
0926 04:07:57.281952 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:57.281963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:57.281997 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:07:57.282124 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:57.282130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:57.282194 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:57.282338 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:57.282586 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.284536 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.285424 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.285885 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.286001 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:57.286726 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:57.286730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:57.286731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:07:57.286940 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:57.286982 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:07:57.286997 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:57.287260 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:57.287294 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 04:07:57.287376 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260926 04:07:57.288923 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:07:57.292443 6346 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:57.293135 6346 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:57.299257 6346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63d3779f843c47f7a6eb54375e59e1cd: Generated new TSK 0
I20260926 04:07:57.299480 6346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:07:57.461524 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:57.309656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.0.213.254:36453)
0926 04:07:57.309703 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:57.309705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:57.309803 (+ 98us) server_negotiation.cc:407] Connection header received
0926 04:07:57.309884 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:57.309885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:57.309895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:57.309960 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0926 04:07:57.310103 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.310625 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.311829 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.312158 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.312233 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:57.312431 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 04:07:57.461282 (+148851us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 04:07:57.461458 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34654: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 04:07:57.461591 6338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34654: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 04:07:57.461757 6363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:57.309557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36453 (local address 127.0.0.1:34654)
0926 04:07:57.309713 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:57.309729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:57.309816 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:57.309973 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:57.309974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:57.309976 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 04:07:57.310041 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:57.310043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.310724 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.310725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:57.311720 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:57.311724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:57.312257 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:57.312259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:57.312323 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:57.312330 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 04:07:57.461458 (+149128us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 04:07:57.461623 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.213.254:36453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260926 04:07:57.462098 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6267
2026-09-26T04:07:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (331 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:57 dist-test-slave-xf24 krb5kdc[6375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:57 dist-test-slave-xf24 krb5kdc[6375](info): set up 2 sockets
Sep 26 04:07:57 dist-test-slave-xf24 krb5kdc[6375](info): commencing operation
krb5kdc: starting...
W20260926 04:07:58.680382 855 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:07:58 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:07:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:07:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:07:58.811618 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:44625
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:44625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:58.893232 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:58.893450 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:58.893473 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:58.894580 6388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:07:58.894616 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:58.894630 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:58.894645 6388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:07:58.894657 6388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:07:58.896148 6388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:44625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:58.896347 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:58.896519 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:58.901401 6394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:58.901552 6396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:58.901577 6388 server_base.cc:1034] running on GCE node
W20260926 04:07:58.901448 6393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:58.901958 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:58.902254 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:58.903458 6388 hybrid_clock.cc:648] HybridClock initialized: now 1790395678903443 us; error 27 us; skew 500 ppm
Sep 26 04:07:58 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:58.906342 6388 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:07:58.906746 6388 webserver.cc:460] Webserver started at http://127.0.213.254:40859/ using document root <none> and password file <none>
I20260926 04:07:58.906936 6388 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:58.906978 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:58.907065 6388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:58.908176 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "66255d02bb5b461da673cd619796fb2e"
format_stamp: "Formatted at 2026-09-26 04:07:58 on dist-test-slave-xf24"
server_key: "511e331fdfb8a91710cb7544a665c8fa"
server_key_iv: "937978359ddf4500af0bf4458a4ab69d"
server_key_version: "encryptionkey@0"
I20260926 04:07:58.908491 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "66255d02bb5b461da673cd619796fb2e"
format_stamp: "Formatted at 2026-09-26 04:07:58 on dist-test-slave-xf24"
server_key: "511e331fdfb8a91710cb7544a665c8fa"
server_key_iv: "937978359ddf4500af0bf4458a4ab69d"
server_key_version: "encryptionkey@0"
I20260926 04:07:58.909780 6388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:58.910534 6403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:58.910704 6388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:58.910782 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "66255d02bb5b461da673cd619796fb2e"
format_stamp: "Formatted at 2026-09-26 04:07:58 on dist-test-slave-xf24"
server_key: "511e331fdfb8a91710cb7544a665c8fa"
server_key_iv: "937978359ddf4500af0bf4458a4ab69d"
server_key_version: "encryptionkey@0"
I20260926 04:07:58.910883 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:58.918720 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:58.921619 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:58.921758 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:58.926620 6388 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:44625
I20260926 04:07:58.927081 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:44625 every 8 connection(s)
I20260926 04:07:58.927191 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:07:58.928661 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:07:58.931806 6456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e: Bootstrap starting.
I20260926 04:07:58.932626 6456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e: Neither blocks nor log segments found. Creating new log.
I20260926 04:07:58.932963 6456 log.cc:824] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:07:58.933720 6456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e: No bootstrap required, opened a new log
I20260926 04:07:58.935272 6456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66255d02bb5b461da673cd619796fb2e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44625 } }
I20260926 04:07:58.935413 6456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:07:58.935436 6456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66255d02bb5b461da673cd619796fb2e, State: Initialized, Role: FOLLOWER
I20260926 04:07:58.935590 6456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66255d02bb5b461da673cd619796fb2e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44625 } }
I20260926 04:07:58.935671 6456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:07:58.935695 6456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:07:58.935722 6456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:07:58.936542 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6388
I20260926 04:07:58.936661 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:07:58.936707 6456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66255d02bb5b461da673cd619796fb2e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44625 } }
I20260926 04:07:58.936837 6456 leader_election.cc:302] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66255d02bb5b461da673cd619796fb2e; no voters:
I20260926 04:07:58.936900 855 external_mini_cluster.cc:1428] Setting key 7b341935f592833d3ae15f6e8c4fe2d0
I20260926 04:07:58.937032 6456 leader_election.cc:288] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:07:58.937083 6459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:07:58.937265 6459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [term 1 LEADER]: Becoming Leader. State: Replica: 66255d02bb5b461da673cd619796fb2e, State: Running, Role: LEADER
I20260926 04:07:58.937310 6456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:07:58.937414 6459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66255d02bb5b461da673cd619796fb2e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44625 } }
I20260926 04:07:58.937965 6460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66255d02bb5b461da673cd619796fb2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66255d02bb5b461da673cd619796fb2e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44625 } } }
I20260926 04:07:58.938060 6460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:58.937948 6459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66255d02bb5b461da673cd619796fb2e. Latest consensus state: current_term: 1 leader_uuid: "66255d02bb5b461da673cd619796fb2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66255d02bb5b461da673cd619796fb2e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 44625 } } }
I20260926 04:07:58.938369 6459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e [sys.catalog]: This master's current role is: LEADER
I20260926 04:07:58.938274 6470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:07:58.938832 6470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:07:58.940523 6470 catalog_manager.cc:1347] Generated new cluster ID: d744e1d0a12f49fd84046f497b5446b3
I20260926 04:07:58.940573 6470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:07:58 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:58.950599 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:58.937558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.0.213.254:44625)
0926 04:07:58.939507 (+ 1949us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:58.939516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:58.939548 (+ 32us) server_negotiation.cc:407] Connection header received
0926 04:07:58.939678 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:58.939685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:58.939752 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:58.939915 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:58.940732 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:58.942183 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:58.943213 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:58.943859 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:58.943981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:58.946765 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:58.946769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:58.946770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:58.946790 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:58.948864 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:58.949275 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:58.949278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:58.949279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:58.949407 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:58.949609 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:58.949610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:58.949611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:58.949948 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:58.950104 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:58.950182 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 04:07:58.950328 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1902,"thread_start_us":61,"threads_started":1}
I20260926 04:07:58.952232 6470 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:07:58.952881 6470 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:07:58.955677 6470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66255d02bb5b461da673cd619796fb2e: Generated new TSK 0
I20260926 04:07:58.955879 6470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:07:58.982801 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:44625
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:59.069339 6482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:59.069573 6482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:59.069600 6482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:59.070845 6482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:59.070883 6482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:59.070912 6482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:07:59.072501 6482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:44625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:59.072712 6482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:59.072899 6482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:59.078655 6488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:59.078655 6490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:59.078655 6487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:59.078905 6482 server_base.cc:1034] running on GCE node
I20260926 04:07:59.079213 6482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:59.079443 6482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:59.080617 6482 hybrid_clock.cc:648] HybridClock initialized: now 1790395679080603 us; error 48 us; skew 500 ppm
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.083236 6482 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:07:59.083721 6482 webserver.cc:460] Webserver started at http://127.0.213.193:36881/ using document root <none> and password file <none>
I20260926 04:07:59.083926 6482 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:59.083966 6482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:59.084124 6482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:59.085147 6482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "ebd2ddf015604ea79bf51874c833610d"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "1044e7b44ac6c269981e3e201b934ae8"
server_key_iv: "a12bf5f2f113cd62ca11c2b66723e026"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.085464 6482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "ebd2ddf015604ea79bf51874c833610d"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "1044e7b44ac6c269981e3e201b934ae8"
server_key_iv: "a12bf5f2f113cd62ca11c2b66723e026"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.086834 6482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:07:59.087648 6497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.087888 6482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:07:59.087976 6482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "ebd2ddf015604ea79bf51874c833610d"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "1044e7b44ac6c269981e3e201b934ae8"
server_key_iv: "a12bf5f2f113cd62ca11c2b66723e026"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.088034 6482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:59.138568 6482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:59.141163 6482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:59.141294 6482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:59.141486 6482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:59.141834 6482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:59.141873 6482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.141897 6482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:59.141909 6482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.147918 6482 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:35837
I20260926 04:07:59.148000 6610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:35837 every 8 connection(s)
I20260926 04:07:59.148361 6482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:07:59.148723 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6482
I20260926 04:07:59.148838 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:07:59.149051 855 external_mini_cluster.cc:1428] Setting key 3a6ecd9e60ece843b234140a31b960c2
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
I20260926 04:07:59.162088 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.149288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:53963 (local address 127.0.213.254:44625)
0926 04:07:59.149339 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.149341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.149575 (+ 234us) server_negotiation.cc:407] Connection header received
0926 04:07:59.151060 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.151061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.151077 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.151281 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.152436 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.153350 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.154726 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.155102 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.155204 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.157945 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:59.157948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:59.157948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:59.157958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:59.159936 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.160793 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.160794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.160795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.160858 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.161198 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.161199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.161200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.161393 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:59.161473 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:59.161914 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 04:07:59.162004 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:07:59.162348 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.149262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:44625 (local address 127.0.213.193:53963)
0926 04:07:59.149428 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:59.149478 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:59.150864 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:59.151381 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:59.151394 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:59.152021 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:59.152322 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:59.152329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.153482 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.153483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:59.154606 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:59.154610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.155237 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.155239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:59.155364 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.156048 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:59.156111 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:59.157817 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:59.160153 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:59.160196 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:59.160199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:59.160698 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:59.160949 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:59.160950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:59.160950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:59.161074 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:59.161483 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:59.161491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:59.161657 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 04:07:59.161861 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 04:07:59.162070 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260926 04:07:59.163699 6611 heartbeater.cc:347] Connected to a master server at 127.0.213.254:44625
I20260926 04:07:59.163856 6611 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:59.164127 6611 heartbeater.cc:511] Master 127.0.213.254:44625 requested a full tablet report, sending...
I20260926 04:07:59.164882 6420 ts_manager.cc:194] Registered new tserver with Master: ebd2ddf015604ea79bf51874c833610d (127.0.213.193:35837)
I20260926 04:07:59.165966 6420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:53963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:07:59.181458 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:44625
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:59.265326 6616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:59.265628 6616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:59.265673 6616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:59.266932 6616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:59.267009 6616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:59.267052 6616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:07:59.268750 6616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:44625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:59.269024 6616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:59.269271 6616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:59.274569 6624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:59.274742 6621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:59.274569 6622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:59.274813 6616 server_base.cc:1034] running on GCE node
I20260926 04:07:59.275102 6616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:59.275372 6616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:59.276525 6616 hybrid_clock.cc:648] HybridClock initialized: now 1790395679276490 us; error 59 us; skew 500 ppm
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.279336 6616 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:07:59.279860 6616 webserver.cc:460] Webserver started at http://127.0.213.194:40521/ using document root <none> and password file <none>
I20260926 04:07:59.280160 6616 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:59.280234 6616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:59.280398 6616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:59.281518 6616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "b89a26aa9ee34343abc59ddf3a6031c6"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "5c029c875da26cc58e3c1a5a7d4df7eb"
server_key_iv: "34c37d37334a55352dc544a498d49d0b"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.281901 6616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "b89a26aa9ee34343abc59ddf3a6031c6"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "5c029c875da26cc58e3c1a5a7d4df7eb"
server_key_iv: "34c37d37334a55352dc544a498d49d0b"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.283298 6616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:07:59.284181 6631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.284351 6616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:07:59.284452 6616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "b89a26aa9ee34343abc59ddf3a6031c6"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "5c029c875da26cc58e3c1a5a7d4df7eb"
server_key_iv: "34c37d37334a55352dc544a498d49d0b"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.284538 6616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:59.316494 6616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:59.319017 6616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:59.319196 6616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:59.319463 6616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:59.319808 6616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:59.319869 6616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.319909 6616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:59.319939 6616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.325688 6616 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:42881
I20260926 04:07:59.325721 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:42881 every 8 connection(s)
I20260926 04:07:59.326177 6616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:07:59.327014 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6616
I20260926 04:07:59.327137 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:07:59.327359 855 external_mini_cluster.cc:1428] Setting key 7628b6ad778846efa41630705767ddc1
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
I20260926 04:07:59.338666 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.327811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:57489 (local address 127.0.213.254:44625)
0926 04:07:59.327863 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.327866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.328128 (+ 262us) server_negotiation.cc:407] Connection header received
0926 04:07:59.329133 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.329134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.329146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.329261 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.330463 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.330973 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.332638 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.332828 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.332905 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.335457 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:59.335459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:59.335459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:59.335468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:59.337136 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.337551 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.337553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.337553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.337612 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.337937 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.337938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.337939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.338076 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:59.338155 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:59.338476 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 04:07:59.338560 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:07:59.338896 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.327830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:44625 (local address 127.0.213.194:57489)
0926 04:07:59.327984 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:59.328024 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:59.328988 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:59.329441 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:59.329447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:59.329976 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:59.330316 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:59.330323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.331077 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.331078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:59.332505 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:59.332508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.332926 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.332928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:59.333093 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.333684 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:59.333696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:59.335350 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:59.337227 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:59.337234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:59.337236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:59.337462 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:59.337684 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:59.337685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:59.337685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:59.337866 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:59.338159 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:59.338165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:59.338304 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 04:07:59.338445 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 04:07:59.338606 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260926 04:07:59.339470 6745 heartbeater.cc:347] Connected to a master server at 127.0.213.254:44625
I20260926 04:07:59.339596 6745 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:59.339773 6745 heartbeater.cc:511] Master 127.0.213.254:44625 requested a full tablet report, sending...
I20260926 04:07:59.340201 6420 ts_manager.cc:194] Registered new tserver with Master: b89a26aa9ee34343abc59ddf3a6031c6 (127.0.213.194:42881)
I20260926 04:07:59.340595 6420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:57489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:07:59.357879 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:44625
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:07:59.440624 6750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:07:59.440927 6750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:07:59.440953 6750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:07:59.442157 6750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:07:59.442200 6750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:07:59.442227 6750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:07:59.443759 6750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:44625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:07:59.444020 6750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:07:59.444326 6750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:07:59.449528 6755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:59.449550 6756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:07:59.449529 6758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:07:59.450093 6750 server_base.cc:1034] running on GCE node
I20260926 04:07:59.450341 6750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:07:59.450661 6750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:07:59.451857 6750 hybrid_clock.cc:648] HybridClock initialized: now 1790395679451846 us; error 28 us; skew 500 ppm
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.454547 6750 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:07:59.455051 6750 webserver.cc:460] Webserver started at http://127.0.213.195:36091/ using document root <none> and password file <none>
I20260926 04:07:59.455309 6750 fs_manager.cc:359] Metadata directory not provided
I20260926 04:07:59.455379 6750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:07:59.455538 6750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:07:59.456765 6750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "0fbf21af151c4e168f5673ce75026e05"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "aec20a5475255f2c8470e2cda2a541b4"
server_key_iv: "cc3a48b07dc22bc37b8ae3d8d9b2a066"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.457121 6750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "0fbf21af151c4e168f5673ce75026e05"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "aec20a5475255f2c8470e2cda2a541b4"
server_key_iv: "cc3a48b07dc22bc37b8ae3d8d9b2a066"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.458451 6750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 04:07:59.459246 6765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.459424 6750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.459527 6750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "0fbf21af151c4e168f5673ce75026e05"
format_stamp: "Formatted at 2026-09-26 04:07:59 on dist-test-slave-xf24"
server_key: "aec20a5475255f2c8470e2cda2a541b4"
server_key_iv: "cc3a48b07dc22bc37b8ae3d8d9b2a066"
server_key_version: "encryptionkey@0"
I20260926 04:07:59.459615 6750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:07:59.484274 6750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:07:59.486742 6750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:07:59.486918 6750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:07:59.487186 6750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:07:59.487617 6750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:07:59.487673 6750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.487707 6750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:07:59.487749 6750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:07:59.493906 6750 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:44309
I20260926 04:07:59.493949 6878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:44309 every 8 connection(s)
I20260926 04:07:59.494320 6750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:07:59.503604 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6750
I20260926 04:07:59.503762 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:07:59.503975 855 external_mini_cluster.cc:1428] Setting key 84e8207e5f0f7506ae5ac8e7888f6b9e
I20260926 04:07:59.504908 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.495125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34589 (local address 127.0.213.254:44625)
0926 04:07:59.495199 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.495201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.495427 (+ 226us) server_negotiation.cc:407] Connection header received
0926 04:07:59.496481 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.496481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.496493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.496563 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.497288 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.497860 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.499126 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.499410 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.499478 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.501824 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:59.501826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:59.501826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:59.501834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:59.503218 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.503684 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.503685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.503685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.503761 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.504035 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.504036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.504036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.504197 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:59.504389 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:59.504724 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 04:07:59.504813 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:07:59.505146 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.495094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:44625 (local address 127.0.213.195:34589)
0926 04:07:59.495279 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 04:07:59.495325 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:07:59.496372 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:07:59.496639 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:07:59.496643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:07:59.496971 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:07:59.497169 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:59.497174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.497996 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.498000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:59.498899 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:07:59.498901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.499526 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.499527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:07:59.499654 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.500284 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:07:59.500293 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:07:59.501715 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:07:59.503339 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:59.503348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:59.503351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:59.503596 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:59.503852 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:07:59.503853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:07:59.503853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:07:59.503962 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:07:59.504390 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:07:59.504397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:07:59.504571 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 04:07:59.504699 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 04:07:59.504872 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260926 04:07:59.505584 6879 heartbeater.cc:347] Connected to a master server at 127.0.213.254:44625
I20260926 04:07:59.505702 6879 heartbeater.cc:464] Registering TS with master...
I20260926 04:07:59.505863 6879 heartbeater.cc:511] Master 127.0.213.254:44625 requested a full tablet report, sending...
I20260926 04:07:59.506310 6420 ts_manager.cc:194] Registered new tserver with Master: 0fbf21af151c4e168f5673ce75026e05 (127.0.213.195:44309)
I20260926 04:07:59.506726 6420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:34589
I20260926 04:07:59.516847 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/test-admin.keytab.
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.538175 855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 04:07:59.540102 6886 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:59.539237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.0.213.193:35837)
0926 04:07:59.539411 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.539417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.539429 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:07:59.539505 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.539507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.539524 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.539613 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.539995 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36506: BlockingRecv error: recv got EOF from 127.0.0.1:36506 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20260926 04:07:59.540201 855 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc' of type FILE
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.541436 855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.542650 855 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:59.553455 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.543296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.213.193:35837)
0926 04:07:59.543340 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.543341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.543404 (+ 63us) server_negotiation.cc:407] Connection header received
0926 04:07:59.543497 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.543498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.543509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.543555 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.543971 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.545442 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.546255 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.546866 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.546932 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.549648 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:59.549658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:59.549659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:59.549668 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:59.551818 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.552388 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.552392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.552393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.552466 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.552768 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.552769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.552769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.552965 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:59.553162 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:59.553218 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 04:07:59.553319 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:07:59.555585 6886 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:59.554892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.213.193:35837)
0926 04:07:59.554934 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.554936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.554948 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:07:59.555021 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.555022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.555035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.555092 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.555524 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36518: BlockingRecv error: recv got EOF from 127.0.0.1:36518 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:07:59.555585 855 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc' of type FILE
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.557015 855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.558122 855 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:59.566187 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.558740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.213.193:35837)
0926 04:07:59.558779 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.558781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.558885 (+ 104us) server_negotiation.cc:407] Connection header received
0926 04:07:59.558956 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.558957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.558968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.559024 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.559422 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.559931 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.560703 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.560953 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.561029 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.563254 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:59.563255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:59.563256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:59.563263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:59.564756 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.565143 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.565144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.565145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.565207 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.565609 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.565610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.565610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.565731 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:59.565844 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:59.565973 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 04:07:59.566062 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 04:07:59.567909 6886 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:07:59.567325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.0.213.193:35837)
0926 04:07:59.567386 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.567388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.567445 (+ 57us) server_negotiation.cc:407] Connection header received
0926 04:07:59.567517 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.567518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.567527 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.567589 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.567857 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36534: BlockingRecv error: recv got EOF from 127.0.0.1:36534 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 04:07:59.568281 855 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395654118158-855-0/krb5kdc/krb5cc)
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.569474 855 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:07:59.570436 855 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:07:59.578864 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:07:59.571400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.213.193:35837)
0926 04:07:59.571463 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:07:59.571465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:07:59.571475 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:07:59.571502 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:07:59.571503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:07:59.571512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:07:59.571586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:07:59.572006 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.572566 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.573286 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:07:59.573491 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:07:59.573570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:07:59.575822 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:07:59.575824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:07:59.575825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:07:59.575833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:07:59.577440 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.577872 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.577874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.577874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.577934 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:07:59.578147 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:07:59.578148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:07:59.578149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:07:59.578286 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:07:59.578376 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 04:07:59.578639 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 04:07:59.578730 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:07:59.579567 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6482
I20260926 04:07:59.584437 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6616
I20260926 04:07:59.588792 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6750
I20260926 04:07:59.592717 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6388
2026-09-26T04:07:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1954 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6899](info): set up 2 sockets
Sep 26 04:07:59 dist-test-slave-xf24 krb5kdc[6899](info): commencing operation
krb5kdc: starting...
W20260926 04:08:00.652460 855 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:08:00 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab.
Entry for principal oryx/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab.
I20260926 04:08:00.775792 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46751
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab
--principal=oryx/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:00.857957 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:00.858202 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:00.858238 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:00.859447 6912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:00.859519 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:00.859540 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:00.859560 6912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:00.859592 6912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:00.861217 6912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab
--principal=oryx/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:00.861513 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:00.861749 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:00.866704 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:00.866709 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:00.866716 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:00.866770 6912 server_base.cc:1034] running on GCE node
I20260926 04:08:00.867182 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:00.867480 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:00.868650 6912 hybrid_clock.cc:648] HybridClock initialized: now 1790395680868618 us; error 42 us; skew 500 ppm
Sep 26 04:08:00 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:00.879732 6912 init.cc:377] Logged in from keytab as oryx/127.0.213.254@KRBTEST.COM (short username oryx)
I20260926 04:08:00.880362 6912 webserver.cc:460] Webserver started at http://127.0.213.254:37317/ using document root <none> and password file <none>
I20260926 04:08:00.880633 6912 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:00.880684 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:00.880836 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:00.881953 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "6022f2cb9cc14142b3e639b0c99783ca"
format_stamp: "Formatted at 2026-09-26 04:08:00 on dist-test-slave-xf24"
server_key: "595ebfa43d291dcccfa7033bd07c784c"
server_key_iv: "bbd022817be933977ac2f70803390e03"
server_key_version: "encryptionkey@0"
I20260926 04:08:00.882328 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "6022f2cb9cc14142b3e639b0c99783ca"
format_stamp: "Formatted at 2026-09-26 04:08:00 on dist-test-slave-xf24"
server_key: "595ebfa43d291dcccfa7033bd07c784c"
server_key_iv: "bbd022817be933977ac2f70803390e03"
server_key_version: "encryptionkey@0"
I20260926 04:08:00.883759 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:00.884651 6927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:00.884835 6912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:08:00.884923 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "6022f2cb9cc14142b3e639b0c99783ca"
format_stamp: "Formatted at 2026-09-26 04:08:00 on dist-test-slave-xf24"
server_key: "595ebfa43d291dcccfa7033bd07c784c"
server_key_iv: "bbd022817be933977ac2f70803390e03"
server_key_version: "encryptionkey@0"
I20260926 04:08:00.885001 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:00.901396 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:00.903764 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:00.903882 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:00.907545 6912 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:46751
I20260926 04:08:00.907616 6979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:46751 every 8 connection(s)
I20260926 04:08:00.907955 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:00.908972 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:00.911044 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 6912
I20260926 04:08:00.911144 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:08:00.911372 6980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Bootstrap starting.
I20260926 04:08:00.911381 855 external_mini_cluster.cc:1428] Setting key 7374958e170337e6e58d2911fa565266
I20260926 04:08:00.912020 6980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:00.912374 6980 log.cc:824] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:00.913146 6980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: No bootstrap required, opened a new log
I20260926 04:08:00.915434 6980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } }
I20260926 04:08:00.915571 6980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:00.915618 6980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6022f2cb9cc14142b3e639b0c99783ca, State: Initialized, Role: FOLLOWER
I20260926 04:08:00.915786 6980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } }
I20260926 04:08:00.915910 6980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:00.916153 6980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:00.916218 6980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:00.916913 6980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } }
I20260926 04:08:00.917034 6980 leader_election.cc:302] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6022f2cb9cc14142b3e639b0c99783ca; no voters:
I20260926 04:08:00.917203 6980 leader_election.cc:288] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:08:00.917255 6985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:00.917461 6980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:00.917788 6985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 LEADER]: Becoming Leader. State: Replica: 6022f2cb9cc14142b3e639b0c99783ca, State: Running, Role: LEADER
I20260926 04:08:00.917909 6985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } }
I20260926 04:08:00.918308 6987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } }
I20260926 04:08:00.918401 6987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:00.918567 6986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6022f2cb9cc14142b3e639b0c99783ca. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } }
I20260926 04:08:00.918640 6986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:00.918753 6996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 04:08:00 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:00.919574 6996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:00.921468 6996 catalog_manager.cc:1347] Generated new cluster ID: 3f249b536a1740ce9f1292cd6e015d84
I20260926 04:08:00.921521 6996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:08:00.923380 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:00.911962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.0.213.254:46751)
0926 04:08:00.912208 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:00.912218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:00.912254 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:08:00.912784 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:00.912813 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:00.912877 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:00.913035 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:00.913719 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:00.915158 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:00.916369 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:00.916994 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:00.917113 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:00.920029 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:00.920033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:00.920033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:00.920044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:00.921919 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:00.922282 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:00.922285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:00.922285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:00.922376 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:00.922588 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:00.922589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:00.922589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:00.922820 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:00.922992 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:00.923070 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 04:08:00.923202 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":120,"threads_started":1}
I20260926 04:08:00.937770 6996 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:08:00.938500 6996 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:00.947005 6996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Generated new TSK 0
I20260926 04:08:00.947227 6996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab.
Entry for principal oryx/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab.
I20260926 04:08:00.982430 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46751
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:01.081907 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:01.082124 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:01.082149 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:01.083339 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:01.083381 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:01.083408 7006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:01.084890 7006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:01.085155 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:01.085397 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:08:01.090780 7006 server_base.cc:1034] running on GCE node
W20260926 04:08:01.090731 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.090739 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.090728 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.091187 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:01.091399 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:01.092545 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790395681092532 us; error 32 us; skew 500 ppm
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:01.095170 7006 init.cc:377] Logged in from keytab as oryx/127.0.213.193@KRBTEST.COM (short username oryx)
I20260926 04:08:01.095700 7006 webserver.cc:460] Webserver started at http://127.0.213.193:43757/ using document root <none> and password file <none>
I20260926 04:08:01.095950 7006 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.096024 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.096223 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:01.097273 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "c5152c40fae1409c96230d6c1d9d53a5"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d6a8e0e967a7a5dc4dffaf075c8f28e0"
server_key_iv: "503fcac34cf1d6a29443a636c93b5f22"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.097635 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "c5152c40fae1409c96230d6c1d9d53a5"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d6a8e0e967a7a5dc4dffaf075c8f28e0"
server_key_iv: "503fcac34cf1d6a29443a636c93b5f22"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.099018 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:01.099758 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.099988 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:01.100127 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "c5152c40fae1409c96230d6c1d9d53a5"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d6a8e0e967a7a5dc4dffaf075c8f28e0"
server_key_iv: "503fcac34cf1d6a29443a636c93b5f22"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.100219 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.110422 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:01.113261 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:01.113430 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:01.113685 7006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:01.114017 7006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:01.114073 7006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.114117 7006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:01.114152 7006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.120383 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:41619
I20260926 04:08:01.120486 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:41619 every 8 connection(s)
I20260926 04:08:01.120802 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:01.128201 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7006
I20260926 04:08:01.128307 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:08:01.128515 855 external_mini_cluster.cc:1428] Setting key fc82cac34d8d8ff667d5852d76a502ca
I20260926 04:08:01.131381 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.121573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:59639 (local address 127.0.213.254:46751)
0926 04:08:01.121621 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.121623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.121847 (+ 224us) server_negotiation.cc:407] Connection header received
0926 04:08:01.122375 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.122377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.122389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.122448 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.123458 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.124007 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.125088 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.125298 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.125386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.127858 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.127860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.127860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.127869 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.129763 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.130295 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.130296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.130296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.130350 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.130673 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.130674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.130674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.130831 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.130907 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.131171 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.131270 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:08:01.131422 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.121545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:59639)
0926 04:08:01.121722 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.121759 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.122271 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.122524 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.122528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.123119 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.123328 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.123333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.124146 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.124147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.124977 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.124979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.125397 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.125398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.125544 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.126176 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.126186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.127740 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.129853 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.129863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.129866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.130215 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.130421 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.130422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.130422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.130597 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.130902 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.130907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.131038 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.131126 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.131252 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260926 04:08:01.132004 7135 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46751
I20260926 04:08:01.132149 7135 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:01.132339 7135 heartbeater.cc:511] Master 127.0.213.254:46751 requested a full tablet report, sending...
I20260926 04:08:01.132961 6944 ts_manager.cc:194] Registered new tserver with Master: c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619)
I20260926 04:08:01.133967 6944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:59639
WARNING: no policy specified for oryx/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab.
Entry for principal oryx/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab.
I20260926 04:08:01.155471 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46751
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:01.239138 7140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:01.239383 7140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:01.239410 7140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:01.240607 7140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:01.240649 7140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:01.240676 7140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:01.242130 7140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:01.242331 7140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:01.242514 7140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:01.247807 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.247797 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.247793 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.248121 7140 server_base.cc:1034] running on GCE node
I20260926 04:08:01.248405 7140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:01.248667 7140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:01.249831 7140 hybrid_clock.cc:648] HybridClock initialized: now 1790395681249808 us; error 45 us; skew 500 ppm
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:01.252403 7140 init.cc:377] Logged in from keytab as oryx/127.0.213.194@KRBTEST.COM (short username oryx)
I20260926 04:08:01.252897 7140 webserver.cc:460] Webserver started at http://127.0.213.194:37807/ using document root <none> and password file <none>
I20260926 04:08:01.253105 7140 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.253157 7140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.253252 7140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:01.254281 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "69955364d6cb442087ab54c46c4a016a"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d19961d8d009130886404d31b9db7301"
server_key_iv: "eecb893d31fcd0ed0488ee0f8d1a65db"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.254599 7140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "69955364d6cb442087ab54c46c4a016a"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d19961d8d009130886404d31b9db7301"
server_key_iv: "eecb893d31fcd0ed0488ee0f8d1a65db"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.255921 7140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:01.256824 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.257053 7140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:01.257139 7140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "69955364d6cb442087ab54c46c4a016a"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d19961d8d009130886404d31b9db7301"
server_key_iv: "eecb893d31fcd0ed0488ee0f8d1a65db"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.257201 7140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.266109 7140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:01.268453 7140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:01.268570 7140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:01.268751 7140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:01.269060 7140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:01.269096 7140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.269119 7140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:01.269131 7140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.274799 7140 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:45885
I20260926 04:08:01.274873 7268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:45885 every 8 connection(s)
I20260926 04:08:01.275213 7140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:08:01.280687 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7140
I20260926 04:08:01.280779 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:08:01.281011 855 external_mini_cluster.cc:1428] Setting key fbb34bf2fa233922ac6a671b93f1592b
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:01.286016 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.276049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:48147 (local address 127.0.213.254:46751)
0926 04:08:01.276156 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.276159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.276368 (+ 209us) server_negotiation.cc:407] Connection header received
0926 04:08:01.276922 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.276923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.276935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.276994 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.278042 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.278548 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.279627 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.279827 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.279912 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.282750 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.282751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.282752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.282760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.284498 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.284906 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.284908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.284908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.284982 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.285246 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.285247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.285248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.285410 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.285488 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.285805 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.285895 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:08:01.286106 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.276026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:48147)
0926 04:08:01.276222 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.276270 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.276807 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.277078 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.277082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.277704 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.277929 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.277934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.278643 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.278644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.279519 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.279522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.279910 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.279911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.280048 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.281054 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.281067 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.282641 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.284587 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.284593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.284595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.284827 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.285065 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.285066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.285066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.285174 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.285501 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.285508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.285662 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.285755 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.285909 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260926 04:08:01.286625 7269 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46751
I20260926 04:08:01.286734 7269 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:01.286914 7269 heartbeater.cc:511] Master 127.0.213.254:46751 requested a full tablet report, sending...
I20260926 04:08:01.287351 6944 ts_manager.cc:194] Registered new tserver with Master: 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885)
I20260926 04:08:01.287927 6944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:48147
WARNING: no policy specified for oryx/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab.
Entry for principal oryx/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab.
I20260926 04:08:01.311163 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46751
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:01.394515 7274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:01.394778 7274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:01.394819 7274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:01.396124 7274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:01.396193 7274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:01.396240 7274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:01.397804 7274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:01.398092 7274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:01.398332 7274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:01.403694 7279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.403717 7282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.403728 7280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.404114 7274 server_base.cc:1034] running on GCE node
I20260926 04:08:01.404320 7274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:01.404588 7274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:01.405761 7274 hybrid_clock.cc:648] HybridClock initialized: now 1790395681405742 us; error 51 us; skew 500 ppm
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:01.408509 7274 init.cc:377] Logged in from keytab as oryx/127.0.213.195@KRBTEST.COM (short username oryx)
I20260926 04:08:01.408943 7274 webserver.cc:460] Webserver started at http://127.0.213.195:36281/ using document root <none> and password file <none>
I20260926 04:08:01.409181 7274 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.409227 7274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.409369 7274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:01.410496 7274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "cb6382895bb340558b73b8549a65e4ec"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "6d8eb7bc1403dd747462417a39296f71"
server_key_iv: "8e5d2b516145ed0290ba66b5d9cfa6ed"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.410899 7274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "cb6382895bb340558b73b8549a65e4ec"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "6d8eb7bc1403dd747462417a39296f71"
server_key_iv: "8e5d2b516145ed0290ba66b5d9cfa6ed"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.412357 7274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:01.413465 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.413693 7274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:01.413797 7274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "cb6382895bb340558b73b8549a65e4ec"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "6d8eb7bc1403dd747462417a39296f71"
server_key_iv: "8e5d2b516145ed0290ba66b5d9cfa6ed"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.413878 7274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.441392 7274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:01.443876 7274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:01.444031 7274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:01.444324 7274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:01.444649 7274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:01.444684 7274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.444710 7274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:01.444722 7274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.451124 7274 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:37477
I20260926 04:08:01.451223 7402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:37477 every 8 connection(s)
I20260926 04:08:01.451545 7274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:08:01.456634 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7274
I20260926 04:08:01.456734 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:08:01.456907 855 external_mini_cluster.cc:1428] Setting key 47a49d963e29f75e5e486b501303455b
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:01.461763 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.452361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:53145 (local address 127.0.213.254:46751)
0926 04:08:01.452428 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.452430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.452674 (+ 244us) server_negotiation.cc:407] Connection header received
0926 04:08:01.453223 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.453224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.453234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.453295 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.454250 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.454759 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.455791 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.455989 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.456119 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.458733 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.458734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.458734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.458741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.460354 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.460855 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.460857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.460857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.460913 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.461135 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.461135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.461136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.461238 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.461286 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.461630 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.461694 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:08:01.461951 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.452332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:53145)
0926 04:08:01.452496 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.452575 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.453077 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.453362 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.453366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.453949 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.454151 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.454155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.454848 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.454849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.455673 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.455676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.456125 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.456126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.456248 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.456914 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.456923 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.458624 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.460525 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.460532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.460534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.460781 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.460983 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.460984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.460984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.461087 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.461292 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.461296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.461457 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.461604 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.461743 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260926 04:08:01.462473 7403 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46751
I20260926 04:08:01.462584 7403 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:01.462785 7403 heartbeater.cc:511] Master 127.0.213.254:46751 requested a full tablet report, sending...
I20260926 04:08:01.463176 6944 ts_manager.cc:194] Registered new tserver with Master: cb6382895bb340558b73b8549a65e4ec (127.0.213.195:37477)
I20260926 04:08:01.463582 6944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:53145
I20260926 04:08:01.469055 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.253.keytab.
Entry for principal oryx/127.0.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.253.keytab.
I20260926 04:08:01.495267 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:45441
--webserver_interface=127.0.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.253.keytab
--principal=oryx/127.0.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46751,127.0.213.253:45441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:01.577224 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:01.577488 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:01.577524 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:01.578689 7408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:01.578766 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:01.578789 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:01.578810 7408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:01.578836 7408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:01.580394 7408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46751,127.0.213.253:45441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.253.keytab
--principal=oryx/127.0.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:45441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
W20260926 04:08:01.580547 7408 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.213.254:46751,127.0.213.253:45441'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 04:08:01.580713 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:01.580948 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:01.585943 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.585965 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.586153 7408 server_base.cc:1034] running on GCE node
W20260926 04:08:01.586093 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.586550 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:01.586822 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:01.587966 7408 hybrid_clock.cc:648] HybridClock initialized: now 1790395681587935 us; error 39 us; skew 500 ppm
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:01.590647 7408 init.cc:377] Logged in from keytab as oryx/127.0.213.253@KRBTEST.COM (short username oryx)
I20260926 04:08:01.591109 7408 webserver.cc:460] Webserver started at http://127.0.213.253:39123/ using document root <none> and password file <none>
I20260926 04:08:01.591342 7408 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.591385 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.591575 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:01.592739 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/instance:
uuid: "24f67e30facb4c9a91d8672c0053ddd8"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "e3d401b4deba4364e3e6fb5bc80bc710"
server_key_iv: "e8e30962accfe74e99822f41d22aceb5"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.593109 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal/instance:
uuid: "24f67e30facb4c9a91d8672c0053ddd8"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "e3d401b4deba4364e3e6fb5bc80bc710"
server_key_iv: "e8e30962accfe74e99822f41d22aceb5"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.594555 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:01.595384 7423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.595721 7408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:01.595830 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
uuid: "24f67e30facb4c9a91d8672c0053ddd8"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "e3d401b4deba4364e3e6fb5bc80bc710"
server_key_iv: "e8e30962accfe74e99822f41d22aceb5"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.595904 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.606614 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:01.608831 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:01.609010 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:01.613327 7408 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.253:45441
I20260926 04:08:01.613358 7475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.253:45441 every 8 connection(s)
I20260926 04:08:01.613765 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
I20260926 04:08:01.614773 7476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:01.616411 7476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:08:01.620594 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7408
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.253@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
I20260926 04:08:01.627336 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.617172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.0.213.254:46751)
0926 04:08:01.617251 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.617253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.617570 (+ 317us) server_negotiation.cc:407] Connection header received
0926 04:08:01.618144 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.618145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.618155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.618254 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.618910 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.619419 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.620527 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.620835 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.620945 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.623439 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.623440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.623441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.623449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.625096 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.625502 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.625504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.625505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.625572 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.625886 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.625887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.625888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.626088 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.626168 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.627137 (+ 969us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.627232 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:08:01.627564 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.617143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:49662)
0926 04:08:01.617347 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.617396 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.618019 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.618302 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.618306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.618562 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.618834 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.618839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.619497 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.619498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.620397 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.620400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.620954 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.620956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.621122 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.621742 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.621750 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.623354 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.625173 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.625178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.625180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.625418 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.625680 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.625681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.625681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.625803 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.626812 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.626820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.626975 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.627100 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.627301 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":125,"threads_started":1}
I20260926 04:08:01.628248 7476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:08:01.630303 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.621127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.213.253:45441)
0926 04:08:01.621280 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.621285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.621309 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:08:01.621369 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.621372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.621391 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.621511 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.622114 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.623148 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.624171 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.624599 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.624654 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.627154 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.627161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.627162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.627169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.629094 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.629456 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.629458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.629459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.629519 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.629738 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.629739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.629739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.629900 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.630063 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.630175 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.630237 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.253@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
I20260926 04:08:01.636128 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.628575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52512)
0926 04:08:01.628606 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.628616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.628656 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.628919 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.628920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.629096 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.629174 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.629176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.630008 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.630009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.630763 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.630766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.631216 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.631217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.631285 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.631698 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.631703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.632899 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.635075 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.635076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.635076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.635237 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.635449 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.635451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.635451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.635504 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.635750 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.635750 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.635787 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.635854 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.635954 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 04:08:01.636183 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.628607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.0.213.253:45441)
0926 04:08:01.628779 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.628781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.628790 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:01.628811 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.628812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.628823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.628904 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.629255 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.629929 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.630862 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.631132 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.631217 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.632998 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.632999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.632999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.633006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.634972 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.635318 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.635319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.635319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.635371 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.635573 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.635573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.635574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.635685 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.635760 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.635883 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.636004 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260926 04:08:01.637662 7476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Bootstrap starting.
I20260926 04:08:01.638255 7476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:01.638497 7476 log.cc:824] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:01.639190 7476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: No bootstrap required, opened a new log
I20260926 04:08:01.640712 7476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } }
I20260926 04:08:01.640838 7476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:01.640861 7476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f67e30facb4c9a91d8672c0053ddd8, State: Initialized, Role: FOLLOWER
I20260926 04:08:01.640990 7476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } }
I20260926 04:08:01.641300 7476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:01.641258 7483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } } }
I20260926 04:08:01.641364 7483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:08:01.643150 7494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:08:01.643218 7494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:08:01.644779 7408 master_runner.cc:428] Detected that this master 24f67e30facb4c9a91d8672c0053ddd8 is joining an existing cluster
I20260926 04:08:01.644843 7408 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.213.253:45441
I20260926 04:08:01.653136 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.646297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.213.253:45441)
0926 04:08:01.646352 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.646354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.646361 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:01.646378 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.646379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.646388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.646450 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.646849 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.647561 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.648689 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.648973 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.649053 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.650310 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.650311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.650311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.650317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.651911 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652311 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.652312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.652312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.652363 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652561 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.652561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.652562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.652693 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.652756 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.652929 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.653055 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":37}
I20260926 04:08:01.653136 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.646022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52514)
0926 04:08:01.646123 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.646132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.646174 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.646471 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.646472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.646696 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.646759 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.646761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.647643 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.647644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.648591 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.648593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.649068 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.649069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.649132 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.649619 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.649623 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.650208 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.652015 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.652016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.652225 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.652435 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.652436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.652490 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.652776 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.652777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.652817 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.652897 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.653006 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260926 04:08:01.654055 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.645897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.0.213.254:46751)
0926 04:08:01.646135 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.646137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.646147 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:01.646171 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.646172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.646183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.646243 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.646740 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.647555 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.648848 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.649148 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.649223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.650309 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.650310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.650310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.650318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.652253 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652623 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.652624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.652624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.652675 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652923 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.652925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.652925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.653055 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.653100 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.653880 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.653966 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260926 04:08:01.654345 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.645801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:49678)
0926 04:08:01.645971 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.645982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.646059 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.646259 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.646263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.646585 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.646657 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.646659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.648149 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.648150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.648760 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.648761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.649227 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.649228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.649289 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.649804 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.649809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.650208 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.652347 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.652349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.652560 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.652747 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.652747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.652748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.652831 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.653149 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.653150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.653730 (+ 580us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.654146 (+ 416us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.654246 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.254@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
I20260926 04:08:01.664778 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.656896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.0.213.253:45441)
0926 04:08:01.656940 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.656942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.657137 (+ 195us) server_negotiation.cc:407] Connection header received
0926 04:08:01.657221 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.657222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.657230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.657272 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.657766 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.658218 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.659338 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.659528 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.659581 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.662068 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.662070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.662070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.662078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.663466 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.663911 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.663912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.663913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.663973 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.664211 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.664212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.664212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.664324 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.664402 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.664597 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.664677 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:08:01.664791 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.656828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52518)
0926 04:08:01.657020 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.657068 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.657147 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.657286 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.657289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.657633 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.657693 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.657695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.658296 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.658297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.659244 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.659247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.659599 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.659600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.659683 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.660298 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.660303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.661961 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.663592 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.663601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.663603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.663823 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.664041 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.664042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.664042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.664143 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.664395 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.664402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.664455 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.664557 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.664680 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
I20260926 04:08:01.665688 6944 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.213.253:45441: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.213.253"
port: 45441
}
attrs {
promote: true
}
}
dest_uuid: "6022f2cb9cc14142b3e639b0c99783ca"
cas_config_opid_index: -1
I20260926 04:08:01.666023 6944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } }
I20260926 04:08:01.666487 7503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:01.666769 7450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:01.667588 7503 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.213.253:45441
I20260926 04:08:01.667689 7450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Refusing update from remote peer 6022f2cb9cc14142b3e639b0c99783ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 04:08:01.667752 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:01.667809 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:01.667816 7408 master.cc:537] Master@127.0.213.253:45441 shutting down...
I20260926 04:08:01.667999 7504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 04:08:01.668686 6930 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 04:08:01.669123 6930 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca -> Peer 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253:45441): Couldn't send request to peer 24f67e30facb4c9a91d8672c0053ddd8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 04:08:01.670642 7408 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:01.670753 7408 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:01.670805 7408 tablet_replica.cc:330] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: stopping tablet replica
I20260926 04:08:01.683873 7408 master.cc:552] Master@127.0.213.253:45441 shutdown complete.
I20260926 04:08:01.683944 7408 master_runner.cc:305] Clearing existing system tablet
I20260926 04:08:01.684108 7408 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.684157 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.684937 7408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:08:01.685562 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.685760 7408 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.685830 7408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:08:01.685856 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
uuid: "24f67e30facb4c9a91d8672c0053ddd8"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "e3d401b4deba4364e3e6fb5bc80bc710"
server_key_iv: "e8e30962accfe74e99822f41d22aceb5"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.686622 7408 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:01.688241 7408 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 04:08:01.688310 7408 log.cc:1196] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 04:08:01.688504 7408 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Deleting consensus metadata
I20260926 04:08:01.688627 7408 master_runner.cc:315] Copying system tablet from 127.0.213.254:46751
I20260926 04:08:01.689245 7408 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.213.254:46751
I20260926 04:08:01.696192 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.689579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:49688)
0926 04:08:01.689694 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.689713 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.689769 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.689909 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.689910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.690075 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.690131 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.690132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.691090 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.691091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.691685 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.691688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.692184 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.692185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.692245 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.692879 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.692884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.693191 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.695095 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.695096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.695096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.695295 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.695518 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.695518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.695519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.695575 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.695856 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.695856 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.695896 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.695950 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.696030 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":74,"threads_started":1}
I20260926 04:08:01.696218 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.689650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.0.213.254:46751)
0926 04:08:01.689697 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.689699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.689781 (+ 82us) server_negotiation.cc:407] Connection header received
0926 04:08:01.689813 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.689814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.689824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.689905 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.690198 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.690990 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.691784 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.692110 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.692183 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.693286 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.693287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.693287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.693295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.695001 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.695382 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.695383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.695383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.695448 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.695639 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.695639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.695640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.695789 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.695862 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.695994 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.696120 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:08:01.696916 6964 tablet_copy_service.cc:140] P 6022f2cb9cc14142b3e639b0c99783ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 24f67e30facb4c9a91d8672c0053ddd8 ({username='oryx', principal='oryx/127.0.213.253@KRBTEST.COM'} at 127.0.0.1:49688)
I20260926 04:08:01.697006 6964 tablet_copy_service.cc:161] P 6022f2cb9cc14142b3e639b0c99783ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 24f67e30facb4c9a91d8672c0053ddd8 at {username='oryx', principal='oryx/127.0.213.253@KRBTEST.COM'} at 127.0.0.1:49688: session id = 24f67e30facb4c9a91d8672c0053ddd8-00000000000000000000000000000000
I20260926 04:08:01.697690 6964 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 04:08:01.698374 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 04:08:01.699718 7408 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: tablet copy: Starting download of 0 data blocks...
I20260926 04:08:01.699891 7408 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: tablet copy: Starting download of 1 WAL segments...
I20260926 04:08:01.700879 7408 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 04:08:01.702044 6964 tablet_copy_service.cc:342] P 6022f2cb9cc14142b3e639b0c99783ca: Request end of tablet copy session 24f67e30facb4c9a91d8672c0053ddd8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.213.253@KRBTEST.COM'} at 127.0.0.1:49688
I20260926 04:08:01.702128 6964 tablet_copy_service.cc:433] P 6022f2cb9cc14142b3e639b0c99783ca: ending tablet copy session 24f67e30facb4c9a91d8672c0053ddd8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 24f67e30facb4c9a91d8672c0053ddd8
I20260926 04:08:01.705084 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:01.706386 7516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.706444 7518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.706598 7408 server_base.cc:1034] running on GCE node
W20260926 04:08:01.706781 7515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.706950 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:01.707124 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:01.708253 7408 hybrid_clock.cc:648] HybridClock initialized: now 1790395681708247 us; error 26 us; skew 500 ppm
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:01.709854 7408 init.cc:377] Logged in from keytab as oryx/127.0.213.253@KRBTEST.COM (short username oryx)
I20260926 04:08:01.710178 7408 webserver.cc:460] Webserver started at http://127.0.213.253:38419/ using document root <none> and password file <none>
I20260926 04:08:01.710328 7408 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.710377 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.711057 7408 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:01.711715 7525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.711985 7408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:01.712049 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
uuid: "24f67e30facb4c9a91d8672c0053ddd8"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "e3d401b4deba4364e3e6fb5bc80bc710"
server_key_iv: "e8e30962accfe74e99822f41d22aceb5"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.712150 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.718652 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:01.720613 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:01.723728 7408 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.253:45441
I20260926 04:08:01.724264 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
I20260926 04:08:01.724265 7577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.253:45441 every 8 connection(s)
I20260926 04:08:01.724990 7578 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:08:01.725427 7578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Bootstrap starting.
I20260926 04:08:01.727646 7578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:08:01.727885 7578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: Bootstrap complete.
I20260926 04:08:01.728150 7578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } }
I20260926 04:08:01.728236 7578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 24f67e30facb4c9a91d8672c0053ddd8, State: Initialized, Role: LEARNER
I20260926 04:08:01.728287 7578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } }
I20260926 04:08:01.728494 7578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:01.728473 7581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:01.728538 7581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: LEARNER
I20260926 04:08:01.730332 7592 catalog_manager.cc:1259] Loaded cluster ID: 3f249b536a1740ce9f1292cd6e015d84
I20260926 04:08:01.730381 7592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: loading cluster ID for follower catalog manager: success
I20260926 04:08:01.731415 7592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: acquiring CA information for follower catalog manager: success
I20260926 04:08:01.731949 7592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:08:01.739959 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.732600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.0.213.253:45441)
0926 04:08:01.732868 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.732870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.732878 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:01.732900 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.732901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.732912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.732984 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.733606 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.734565 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.735322 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.735627 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.735708 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.737061 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.737063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.737063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.737070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.738761 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.739144 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.739146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.739146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.739211 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.739428 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.739429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.739429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.739564 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.739680 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.739757 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.739849 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for oryx/127.0.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.252.keytab.
Entry for principal oryx/127.0.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.252.keytab.
I20260926 04:08:01.767930 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:43985
--webserver_interface=127.0.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.252.keytab
--principal=oryx/127.0.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:01.866091 7596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:01.866310 7596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:01.866333 7596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:01.867444 7596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:01.867483 7596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:01.867497 7596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:01.867511 7596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:01.867522 7596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:01.869048 7596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.252.keytab
--principal=oryx/127.0.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:01.869261 7596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:01.869443 7596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:01.874311 7601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.874279 7604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:01.874274 7602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:01.874400 7596 server_base.cc:1034] running on GCE node
I20260926 04:08:01.874881 7596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:01.875142 7596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:01.876288 7596 hybrid_clock.cc:648] HybridClock initialized: now 1790395681876276 us; error 27 us; skew 500 ppm
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:01.878734 7596 init.cc:377] Logged in from keytab as oryx/127.0.213.252@KRBTEST.COM (short username oryx)
I20260926 04:08:01.879123 7596 webserver.cc:460] Webserver started at http://127.0.213.252:40281/ using document root <none> and password file <none>
I20260926 04:08:01.879323 7596 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:01.879364 7596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:01.879480 7596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:01.880542 7596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/instance:
uuid: "19ff866ecffc4ff096f4f72eab95001f"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "647e450b3609342345bae694572451fb"
server_key_iv: "486931182da2c7bde1e4cdd4aaa4ea3e"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.880833 7596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal/instance:
uuid: "19ff866ecffc4ff096f4f72eab95001f"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "647e450b3609342345bae694572451fb"
server_key_iv: "486931182da2c7bde1e4cdd4aaa4ea3e"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.882161 7596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:01.882884 7611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:01.883077 7596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:01.883181 7596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
uuid: "19ff866ecffc4ff096f4f72eab95001f"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "647e450b3609342345bae694572451fb"
server_key_iv: "486931182da2c7bde1e4cdd4aaa4ea3e"
server_key_version: "encryptionkey@0"
I20260926 04:08:01.883256 7596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:01.895610 7596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:01.897863 7596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:01.898033 7596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:01.902782 7596 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.252:43985
I20260926 04:08:01.902851 7663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.252:43985 every 8 connection(s)
I20260926 04:08:01.903201 7596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
I20260926 04:08:01.903919 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7596
I20260926 04:08:01.904480 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:01.906919 7664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.213.252@KRBTEST.COM
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.252@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:01.915475 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.904635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.213.252:43985)
0926 04:08:01.904825 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.904836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.904869 (+ 33us) server_negotiation.cc:407] Connection header received
0926 04:08:01.905353 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.905360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.905413 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.905529 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.906281 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.907211 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.908267 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.908825 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.908945 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.911651 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.911658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.911659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.911666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.913867 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.914217 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.914221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.914221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.914320 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.914514 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.914515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.914515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.914778 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.914945 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.915061 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.915202 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260926 04:08:01.915652 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.907683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.0.213.254:46751)
0926 04:08:01.907727 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.907729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.907890 (+ 161us) server_negotiation.cc:407] Connection header received
0926 04:08:01.907961 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.907962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.907973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.908024 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.908625 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.909346 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.910172 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.910379 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.910463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.913052 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.913054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.913054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.913062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.914523 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.914838 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.914839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.914839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.914896 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.915112 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.915113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.915113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.915243 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.915325 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.915487 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.915564 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:08:01.915688 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.907547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:49700)
0926 04:08:01.907794 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.907824 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.907909 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.908077 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.908081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.908455 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.908530 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.908532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.909451 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.909452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.910061 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.910063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.910491 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.910493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.910579 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.911124 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.911134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.912970 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.914592 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.914597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.914599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.914767 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.914973 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.914974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.914974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.915045 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.915331 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.915336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.915375 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.915446 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.915553 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260926 04:08:01.916415 7664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.252@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
I20260926 04:08:01.922955 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.916766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.0.213.253:45441)
0926 04:08:01.916808 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.916810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.916817 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:01.916839 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.916839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.916850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.916923 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.917183 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.917814 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.918609 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.918824 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.918907 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.920755 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.920757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.920757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.920763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.921999 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.922318 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.922319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.922319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.922367 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.922538 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.922539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.922539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.922634 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.922682 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.922812 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.922891 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:08:01.922960 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.916654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52544)
0926 04:08:01.916704 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.916716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.916757 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.916924 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.916925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.917063 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.917105 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.917106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.917878 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.917879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.918537 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.918539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.918907 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.918907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.919002 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.919423 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.919427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.920657 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.922077 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.922078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.922078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.922247 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.922430 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.922431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.922431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.922484 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.922692 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.922692 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.922728 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.922795 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.922883 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 04:08:01.923472 7664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395681, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.252@KRBTEST.COM for oryx/127.0.213.252@KRBTEST.COM
I20260926 04:08:01.929917 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.923720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.0.1:50926)
0926 04:08:01.923774 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.923785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.923867 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.924023 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.924024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.924191 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.924220 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.924221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.924829 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.924830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.925342 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.925344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.925677 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.925678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.925717 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.926157 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.926161 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.927329 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.928995 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.928996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.928996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.929152 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.929348 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.929349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.929349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.929396 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.929665 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.929666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.929702 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.929759 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.929845 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 04:08:01.930019 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.923750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.0.213.252:43985)
0926 04:08:01.923817 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.923819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.923853 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:08:01.923942 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.923942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.923952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.924010 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.924286 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.924738 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.925420 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.925607 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.925661 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.927462 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.927464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.927464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.927471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.928866 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.929219 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.929220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.929220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.929269 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.929516 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.929517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.929517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.929609 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.929656 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.929773 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.929886 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":50}
I20260926 04:08:01.931330 7664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Bootstrap starting.
I20260926 04:08:01.932237 7664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:01.932528 7664 log.cc:824] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:01.933293 7664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: No bootstrap required, opened a new log
I20260926 04:08:01.934710 7664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } }
I20260926 04:08:01.934859 7664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:01.934906 7664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ff866ecffc4ff096f4f72eab95001f, State: Initialized, Role: FOLLOWER
I20260926 04:08:01.935050 7664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } }
I20260926 04:08:01.935367 7664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:01.935344 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } } }
I20260926 04:08:01.935465 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:08:01.937731 7680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:08:01.937795 7680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:08:01.939963 7596 master_runner.cc:428] Detected that this master 19ff866ecffc4ff096f4f72eab95001f is joining an existing cluster
I20260926 04:08:01.940006 7596 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.213.252:43985
I20260926 04:08:01.947878 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.941080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.0.213.252:43985)
0926 04:08:01.941156 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.941158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.941416 (+ 258us) server_negotiation.cc:407] Connection header received
0926 04:08:01.941515 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.941515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.941525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.941553 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.942010 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.942723 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.943855 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.944148 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.944189 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.945284 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.945285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.945285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.945293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.946811 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.947127 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.947128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.947129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.947188 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.947361 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.947362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.947362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.947499 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.947568 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.947673 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.947788 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":58}
I20260926 04:08:01.947988 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.941129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.0.1:50928)
0926 04:08:01.941369 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.941377 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.941445 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.941610 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.941610 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.941865 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.941930 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.941931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.942804 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.942805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.943762 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.943764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.944276 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.944277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.944332 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.944846 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.944851 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.945185 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.946898 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.946899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.946899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.947059 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.947253 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.947254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.947254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.947302 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.947561 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.947562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.947596 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.947646 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.947727 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":35,"thread_start_us":67,"threads_started":1}
I20260926 04:08:01.967099 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7006
I20260926 04:08:01.973460 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:41619
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:46751
--webserver_port=43757
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:08:01.980520 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.941020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:49702)
0926 04:08:01.941212 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.941223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.941285 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.941465 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.941466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.941715 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.941803 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.941805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.942772 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.942773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.943761 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.943764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.944319 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.944320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.944410 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.976711 (+ 32301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.976720 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.977270 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.979270 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.979271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.979271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.979531 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.979748 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.979749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.979749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.979803 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.980128 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.980129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.980192 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.980259 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.980376 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":37,"thread_start_us":102,"threads_started":1}
I20260926 04:08:01.980822 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.941073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.0.213.254:46751)
0926 04:08:01.941121 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.941124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.941298 (+ 174us) server_negotiation.cc:407] Connection header received
0926 04:08:01.941334 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.941335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.941347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.941407 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.941894 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.942674 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.943860 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.944203 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.944265 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.977425 (+ 33160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.977427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.977428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.977436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.979136 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.979618 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.979620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.979620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.979680 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.979866 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.979867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.979867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.980009 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.980100 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.980648 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.980730 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:01.980962 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.941258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52548)
0926 04:08:01.941515 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.941525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.941587 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.941768 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.941769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.941950 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.942004 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.942005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.942886 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.942886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.943762 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.943764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.944312 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.944312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.944368 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.976763 (+ 32395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.976770 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.977507 (+ 737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.979570 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.979571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.979571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.979831 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.980119 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.980120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.980121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.980192 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.980532 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.980533 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.980583 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.980658 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.980782 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":74,"thread_start_us":62,"threads_started":1}
I20260926 04:08:01.981853 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.941418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.0.213.253:45441)
0926 04:08:01.941462 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.941463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.941568 (+ 105us) server_negotiation.cc:407] Connection header received
0926 04:08:01.941659 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.941660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.941671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.941703 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.942086 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.942800 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.943859 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.944204 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.944244 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.977644 (+ 33400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.977646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.977646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.977656 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.979446 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.979927 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.979929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.979929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.979997 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.980279 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.980279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.980280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.980434 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.980503 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.981587 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.981696 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 04:08:01 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395680, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.254@KRBTEST.COM for oryx/127.0.213.252@KRBTEST.COM
I20260926 04:08:01.992621 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.984217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.0.1:50940)
0926 04:08:01.984278 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.984290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.984334 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.984637 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.984639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.984822 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:01.984875 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.984877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.985848 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.985849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.986719 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.986722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.987275 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.987276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.987350 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.987928 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:01.987934 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:01.989524 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:01.991427 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.991428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.991429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.991650 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.991848 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:01.991848 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:01.991849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:01.991903 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:01.992282 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:01.992283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:01.992331 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 04:08:01.992393 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 04:08:01.992506 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:08:01.992997 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.984407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.0.213.252:43985)
0926 04:08:01.984487 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.984490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.984500 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:01.984539 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.984541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.984554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.984622 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:01.984959 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.985756 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.986814 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.987194 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.987264 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:01.989626 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:01.989628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:01.989628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:01.989635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:01.991331 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.991722 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.991723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.991723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.991773 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:01.991964 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:01.991964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:01.991965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:01.992194 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:01.992256 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:01.992830 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 04:08:01.992908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 04:08:01.993510 6944 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.213.252:43985: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.213.252"
port: 43985
}
attrs {
promote: true
}
}
dest_uuid: "6022f2cb9cc14142b3e639b0c99783ca"
cas_config_opid_index: 5
I20260926 04:08:01.993762 6944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } }
I20260926 04:08:01.994480 7638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:01.995504 7638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Refusing update from remote peer 6022f2cb9cc14142b3e639b0c99783ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 04:08:01.995765 7503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:01.996802 7503 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.213.252:43985
I20260926 04:08:01.996878 7503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:01.997088 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:01.997143 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:01.997586 7596 master.cc:537] Master@127.0.213.252:43985 shutting down...
I20260926 04:08:02.000352 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.994219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52556)
0926 04:08:01.996258 (+ 2039us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:01.996270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:01.996310 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:01.996523 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:01.996524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:01.996526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:01.996572 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.996574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.997870 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.997871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:01.999205 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:01.999225 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.000101 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.000102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.000168 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.000189 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.000242 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.000284 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023}
I20260926 04:08:02.000923 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:01.995707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.0.213.253:45441)
0926 04:08:01.995778 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:01.995780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:01.996362 (+ 582us) server_negotiation.cc:407] Connection header received
0926 04:08:01.996389 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:01.996390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:01.996413 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:01.996465 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:01.996646 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.997454 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.999320 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:01.999887 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:01.999958 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.000029 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.000805 (+ 776us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.000859 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:08:02.001333 7638 raft_consensus.cc:858] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } }
I20260926 04:08:02.001368 7552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 LEARNER]: Refusing update from remote peer 6022f2cb9cc14142b3e639b0c99783ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260926 04:08:02.001421 7638 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } }
I20260926 04:08:02.001556 7504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 04:08:02.001837 7638 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:02.002039 7552 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.003012 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6022f2cb9cc14142b3e639b0c99783ca. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:02.003080 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 04:08:02.003113 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:02.003149 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 04:08:02.003105 7581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6022f2cb9cc14142b3e639b0c99783ca. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.003180 7581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: LEARNER
I20260926 04:08:02.003175 7669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } }
I20260926 04:08:02.003206 7669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 04:08:02.003212 7581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: NON_VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: true } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.003248 7581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: LEARNER
I20260926 04:08:02.003723 7691 mvcc.cc:205] Tried to move back new op lower bound from 7333460709157449728 to 7333460709040144384. Current Snapshot: MvccSnapshot[applied={T|T < 7333460709050142720}]
I20260926 04:08:02.004113 7504 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: attempting to promote NON_VOTER 24f67e30facb4c9a91d8672c0053ddd8 to VOTER
I20260926 04:08:02.004272 7504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } }
I20260926 04:08:02.004885 7552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 LEARNER]: Refusing update from remote peer 6022f2cb9cc14142b3e639b0c99783ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 04:08:02.005002 6931 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca -> Peer 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252:43985): Couldn't send request to peer 19ff866ecffc4ff096f4f72eab95001f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 04:08:02.005182 7700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 04:08:02.005779 7690 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.005911 7552 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.006893 7504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.006965 7504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:02.006979 7696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.007097 7696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:08:02.007159 7504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.007213 7504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:02.009196 7596 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 04:08:02.009298 7596 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 04:08:02.009330 7596 tablet_replica.cc:330] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: stopping tablet replica
I20260926 04:08:02.022452 7596 master.cc:552] Master@127.0.213.252:43985 shutdown complete.
I20260926 04:08:02.022519 7596 master_runner.cc:305] Clearing existing system tablet
I20260926 04:08:02.022650 7596 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:02.022696 7596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:02.023567 7596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:02.024355 7702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.024528 7596 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:02.024583 7596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:02.024606 7596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
uuid: "19ff866ecffc4ff096f4f72eab95001f"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "647e450b3609342345bae694572451fb"
server_key_iv: "486931182da2c7bde1e4cdd4aaa4ea3e"
server_key_version: "encryptionkey@0"
I20260926 04:08:02.025342 7596 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:02.027035 7596 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 04:08:02.027112 7596 log.cc:1196] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 04:08:02.027366 7596 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Deleting consensus metadata
I20260926 04:08:02.027526 7596 master_runner.cc:315] Copying system tablet from 127.0.213.254:46751
I20260926 04:08:02.028213 7596 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.213.254:46751
I20260926 04:08:02.039436 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.028532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:49704)
0926 04:08:02.032166 (+ 3634us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.032182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.032284 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.032465 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.032466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.032681 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.032748 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.032749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.033737 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.033739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.034589 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.034591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.035095 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.035097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.035207 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.035808 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.035820 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.036288 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.038284 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.038285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.038285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.038502 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.038727 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.038728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.038728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.038791 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.039087 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.039087 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.039137 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.039223 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.039328 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3606,"thread_start_us":89,"threads_started":1}
I20260926 04:08:02.039449 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.028678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.0.213.254:46751)
0926 04:08:02.028735 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.028737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.032295 (+ 3558us) server_negotiation.cc:407] Connection header received
0926 04:08:02.032342 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.032343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.032358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.032444 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.032846 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.033634 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.034692 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.035015 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.035094 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.036380 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.036381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.036382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.036389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.038200 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.038587 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.038588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.038589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.038654 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.038868 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.038869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.038870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.039014 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.039090 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.039265 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.039346 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:08:02.040217 6964 tablet_copy_service.cc:140] P 6022f2cb9cc14142b3e639b0c99783ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 19ff866ecffc4ff096f4f72eab95001f ({username='oryx', principal='oryx/127.0.213.252@KRBTEST.COM'} at 127.0.0.1:49704)
I20260926 04:08:02.040282 6964 tablet_copy_service.cc:161] P 6022f2cb9cc14142b3e639b0c99783ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 19ff866ecffc4ff096f4f72eab95001f at {username='oryx', principal='oryx/127.0.213.252@KRBTEST.COM'} at 127.0.0.1:49704: session id = 19ff866ecffc4ff096f4f72eab95001f-00000000000000000000000000000000
I20260926 04:08:02.040864 6964 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 04:08:02.041447 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 04:08:02.042865 7596 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: tablet copy: Starting download of 0 data blocks...
I20260926 04:08:02.043120 7596 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: tablet copy: Starting download of 1 WAL segments...
I20260926 04:08:02.044035 7596 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 04:08:02.045684 6964 tablet_copy_service.cc:342] P 6022f2cb9cc14142b3e639b0c99783ca: Request end of tablet copy session 19ff866ecffc4ff096f4f72eab95001f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.213.252@KRBTEST.COM'} at 127.0.0.1:49704
I20260926 04:08:02.045743 6964 tablet_copy_service.cc:433] P 6022f2cb9cc14142b3e639b0c99783ca: ending tablet copy session 19ff866ecffc4ff096f4f72eab95001f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 19ff866ecffc4ff096f4f72eab95001f
I20260926 04:08:02.048448 7596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:02.049669 7709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.053259 7710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.053313 7712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:02.053577 7596 server_base.cc:1034] running on GCE node
I20260926 04:08:02.053754 7596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:02.053957 7596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:02.055130 7596 hybrid_clock.cc:648] HybridClock initialized: now 1790395682055108 us; error 44 us; skew 500 ppm
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:02.056849 7596 init.cc:377] Logged in from keytab as oryx/127.0.213.252@KRBTEST.COM (short username oryx)
I20260926 04:08:02.057130 7596 webserver.cc:460] Webserver started at http://127.0.213.252:45627/ using document root <none> and password file <none>
I20260926 04:08:02.057251 7596 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:02.057287 7596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:02.058058 7596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:02.058643 7719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.058825 7596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:08:02.058881 7596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
uuid: "19ff866ecffc4ff096f4f72eab95001f"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "647e450b3609342345bae694572451fb"
server_key_iv: "486931182da2c7bde1e4cdd4aaa4ea3e"
server_key_version: "encryptionkey@0"
I20260926 04:08:02.058943 7596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:02.072332 7596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:02.074453 7596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:02.077832 7596 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.252:43985
I20260926 04:08:02.077872 7771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.252:43985 every 8 connection(s)
I20260926 04:08:02.078176 7596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
I20260926 04:08:02.079180 7772 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:08:02.079733 7772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Bootstrap starting.
W20260926 04:08:02.081460 7689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:02.081676 7689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:02.081720 7689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 04:08:02.082243 7772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:08:02.082468 7772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Bootstrap complete.
I20260926 04:08:02.082665 7772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } }
I20260926 04:08:02.082753 7772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19ff866ecffc4ff096f4f72eab95001f, State: Initialized, Role: LEARNER
I20260926 04:08:02.082790 7772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } }
I20260926 04:08:02.083002 7772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:02.083003 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.083220 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: LEARNER
W20260926 04:08:02.083364 7689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:02.083422 7689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:02.083475 7689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:02.084676 7789 catalog_manager.cc:1259] Loaded cluster ID: 3f249b536a1740ce9f1292cd6e015d84
I20260926 04:08:02.084781 7789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: loading cluster ID for follower catalog manager: success
I20260926 04:08:02.085987 7789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: acquiring CA information for follower catalog manager: success
I20260926 04:08:02.086054 7689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:41619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=43757
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:02.086308 7689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:02.086297 7789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:08:02.086647 7689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:02.091944 7791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.092115 7792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.091959 7794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:02.092815 7689 server_base.cc:1034] running on GCE node
I20260926 04:08:02.093133 7689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:02.093478 7689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:02.094661 7689 hybrid_clock.cc:648] HybridClock initialized: now 1790395682094642 us; error 35 us; skew 500 ppm
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:02.097558 7689 init.cc:377] Logged in from keytab as oryx/127.0.213.193@KRBTEST.COM (short username oryx)
I20260926 04:08:02.098093 7689 webserver.cc:460] Webserver started at http://127.0.213.193:43757/ using document root <none> and password file <none>
I20260926 04:08:02.098326 7689 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:02.098416 7689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:02.100013 7689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:02.100783 7801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.101071 7689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:02.101181 7689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "c5152c40fae1409c96230d6c1d9d53a5"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d6a8e0e967a7a5dc4dffaf075c8f28e0"
server_key_iv: "503fcac34cf1d6a29443a636c93b5f22"
server_key_version: "encryptionkey@0"
I20260926 04:08:02.101876 7689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:02.123508 7689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:02.125808 7689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:02.125936 7689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:02.126147 7689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:02.126523 7689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:02.126562 7689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.126585 7689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:02.126597 7689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.133100 7689 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:41619
I20260926 04:08:02.133621 7689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:02.134238 7914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:41619 every 8 connection(s)
I20260926 04:08:02.140161 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7689
I20260926 04:08:02.140334 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7140
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.252@KRBTEST.COM
I20260926 04:08:02.145892 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.134937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:46743 (local address 127.0.213.253:45441)
0926 04:08:02.134983 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.134985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.134995 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:02.135841 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.135842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.135854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.135921 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.136732 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.137239 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.138311 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.138523 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.138602 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.142703 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.142705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.142705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.142714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.144549 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144989 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.144990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.144991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.145049 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.145253 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.145254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.145254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.145406 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.145448 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.145660 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.145754 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:02.146109 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.134457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.213.193:50061)
0926 04:08:02.134630 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.134673 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.135748 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.136014 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.136014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.136476 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.136630 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.136637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.137883 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.137884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.139102 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.139104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.139615 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.139615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.139676 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.140109 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.140114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.142409 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.144441 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.144442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.144651 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.144943 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.144944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.144993 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.145371 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.145372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.145481 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.145632 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.145844 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":8,"thread_start_us":105,"threads_started":1}
I20260926 04:08:02.146246 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.134758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:50061 (local address 127.0.213.252:43985)
0926 04:08:02.134868 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.134871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.134882 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:08:02.135906 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.135907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.135920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.136320 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.136739 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.137793 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.139187 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.139525 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.139811 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.142507 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.142508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.142509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.142516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.144342 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144805 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.144805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.144806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.144865 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.145063 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.145063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.145064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.145199 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.145543 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.146040 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.146133 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260926 04:08:02.146482 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.134987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:55929 (local address 127.0.213.254:46751)
0926 04:08:02.135031 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.135033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.135188 (+ 155us) server_negotiation.cc:407] Connection header received
0926 04:08:02.135864 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.135866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.135878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.135935 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.136741 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.137517 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.139005 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.139319 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.139356 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.141961 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.141963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.141964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.141972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.143886 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144529 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.144531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.144531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.144586 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144922 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.144923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.144923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.145080 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.145133 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.146328 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.146413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:02.146708 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.134457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.213.193:46743)
0926 04:08:02.134630 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.134673 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.135719 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.135998 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.136002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.136421 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.136629 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.136634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.137363 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.137364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.138205 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.138207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.138620 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.138621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.138748 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.139351 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.139360 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.142593 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.144641 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.144642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.144902 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.145117 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.145118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.145119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.145185 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.145491 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.145492 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.145535 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.145631 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.146533 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":137,"thread_start_us":98,"threads_started":1}
I20260926 04:08:02.146885 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.134768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:55929)
0926 04:08:02.135118 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.135126 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.135751 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.136008 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.136009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.136468 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.136630 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.136637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.137611 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.137613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.138905 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.138908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.139431 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.139432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.139490 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.139914 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.139924 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.141819 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.143990 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.143998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.144001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.144442 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.144661 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.144662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.144662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.144833 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.145168 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.145175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.145481 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.146639 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.146736 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260926 04:08:02.147388 7916 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46751
I20260926 04:08:02.147490 7916 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.147677 7916 heartbeater.cc:511] Master 127.0.213.254:46751 requested a full tablet report, sending...
I20260926 04:08:02.148131 6944 ts_manager.cc:194] Re-registered known tserver with Master: c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619)
I20260926 04:08:02.148684 7917 heartbeater.cc:347] Connected to a master server at 127.0.213.253:45441
I20260926 04:08:02.148670 6944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:55929
I20260926 04:08:02.148734 7917 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.148808 7917 heartbeater.cc:511] Master 127.0.213.253:45441 requested a full tablet report, sending...
I20260926 04:08:02.149215 7534 ts_manager.cc:194] Registered new tserver with Master: c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619)
I20260926 04:08:02.151750 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:45885
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:46751
--webserver_port=37807
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:08:02.151875 7915 heartbeater.cc:347] Connected to a master server at 127.0.213.252:43985
I20260926 04:08:02.151927 7915 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.152006 7915 heartbeater.cc:511] Master 127.0.213.252:43985 requested a full tablet report, sending...
I20260926 04:08:02.152426 7736 ts_manager.cc:194] Registered new tserver with Master: c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619)
W20260926 04:08:02.238685 7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:02.238960 7923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:02.239001 7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:02.240337 7923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:02.240409 7923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:02.240458 7923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:02.242588 7923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:45885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=37807
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:02.242939 7923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:02.243222 7923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:02.248761 7929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.248868 7928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:02.248914 7923 server_base.cc:1034] running on GCE node
W20260926 04:08:02.248906 7931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:02.249292 7923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:02.249559 7923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:02.250737 7923 hybrid_clock.cc:648] HybridClock initialized: now 1790395682250719 us; error 53 us; skew 500 ppm
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:02.253396 7923 init.cc:377] Logged in from keytab as oryx/127.0.213.194@KRBTEST.COM (short username oryx)
I20260926 04:08:02.253885 7923 webserver.cc:460] Webserver started at http://127.0.213.194:37807/ using document root <none> and password file <none>
I20260926 04:08:02.254086 7923 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:02.254139 7923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:02.255548 7923 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:02.256213 7938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.256613 7923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:08:02.256722 7923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "69955364d6cb442087ab54c46c4a016a"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "d19961d8d009130886404d31b9db7301"
server_key_iv: "eecb893d31fcd0ed0488ee0f8d1a65db"
server_key_version: "encryptionkey@0"
I20260926 04:08:02.257118 7923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:02.269973 7923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:02.272709 7923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:02.272922 7923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:02.273247 7923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:02.273598 7923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:02.273636 7923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.273660 7923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:02.273674 7923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.279537 7923 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:45885
I20260926 04:08:02.279578 8051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:45885 every 8 connection(s)
I20260926 04:08:02.279954 7923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:02.287247 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 7923
I20260926 04:08:02.287408 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7274
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.252@KRBTEST.COM
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
I20260926 04:08:02.292259 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.280788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:57981 (local address 127.0.213.254:46751)
0926 04:08:02.280842 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.280844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.281151 (+ 307us) server_negotiation.cc:407] Connection header received
0926 04:08:02.281831 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.281833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.281846 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.281908 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.282717 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.283310 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.284602 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.284922 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.284992 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.287695 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.287696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.287697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.287704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.289498 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.290031 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.290032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.290033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.290093 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.291386 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.291387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.291387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.291550 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.291622 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.292015 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.292135 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:08:02.292550 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.281189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:54743 (local address 127.0.213.253:45441)
0926 04:08:02.281231 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.281233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.281316 (+ 83us) server_negotiation.cc:407] Connection header received
0926 04:08:02.281851 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.281853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.281866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.281900 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.282738 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.283500 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.284644 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.284932 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.284992 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.289712 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.289713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.289713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.289722 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.291264 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.291678 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.291679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.291680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.291745 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.291945 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.291946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.291946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.292107 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.292152 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.292352 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.292445 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:08:02.293150 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.280791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:57981)
0926 04:08:02.280967 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.281026 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.281711 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.281974 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.281978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.282406 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.282623 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.282628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.283421 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.283422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.284502 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.284504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.285015 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.285016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.285166 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.285870 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.285875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.287577 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.289598 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.289607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.289610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.289930 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.290413 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.290414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.290414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.290519 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.291651 (+ 1132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.291658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.291848 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.292798 (+ 950us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.292936 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":54,"thread_start_us":93,"threads_started":1}
I20260926 04:08:02.293150 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.281103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.213.194:54743)
0926 04:08:02.281261 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.281268 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.281742 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.282434 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.282435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.282571 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.282623 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.282630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.283601 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.283602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.284580 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.284582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.285050 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.285051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.285165 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.285800 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.285809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.289605 (+ 3796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.291367 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.291369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.291369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.291594 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.291817 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.291817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.291818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.291872 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.292199 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.292200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.292240 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.292668 (+ 428us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.292887 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":33,"thread_start_us":85,"threads_started":1}
I20260926 04:08:02.293499 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.280845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:43747 (local address 127.0.213.252:43985)
0926 04:08:02.280884 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.280886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.281314 (+ 428us) server_negotiation.cc:407] Connection header received
0926 04:08:02.281856 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.281858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.281875 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.281905 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.282747 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.283549 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.284722 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.284929 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.285016 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.289463 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.289464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.289465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.289474 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.292112 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.292562 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.292563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.292563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.292622 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.292816 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.292817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.292817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.292952 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.292993 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.293257 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.293362 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:08:02.293817 8053 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46751
I20260926 04:08:02.294106 8053 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.294200 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:37477
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:46751
--webserver_port=36281
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:08:02.294354 8053 heartbeater.cc:511] Master 127.0.213.254:46751 requested a full tablet report, sending...
I20260926 04:08:02.293886 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.280791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.213.194:43747)
0926 04:08:02.281224 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.281235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.281763 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.281994 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.281995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.282420 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.282624 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.282630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.283656 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.283657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.284571 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.284573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.285410 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.285412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.285472 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.286032 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.286036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.289354 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.292202 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.292203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.292203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.292471 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.292693 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.292694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.292694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.292751 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.293037 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.293038 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.293125 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.293615 (+ 490us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.293729 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":77,"thread_start_us":78,"threads_started":1}
I20260926 04:08:02.294562 8054 heartbeater.cc:347] Connected to a master server at 127.0.213.253:45441
I20260926 04:08:02.294610 8054 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.294682 8054 heartbeater.cc:511] Master 127.0.213.253:45441 requested a full tablet report, sending...
I20260926 04:08:02.294785 6944 ts_manager.cc:194] Re-registered known tserver with Master: 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885)
I20260926 04:08:02.294922 8052 heartbeater.cc:347] Connected to a master server at 127.0.213.252:43985
I20260926 04:08:02.294961 8052 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.295023 8052 heartbeater.cc:511] Master 127.0.213.252:43985 requested a full tablet report, sending...
I20260926 04:08:02.295259 6944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:57981
I20260926 04:08:02.295331 7736 ts_manager.cc:194] Registered new tserver with Master: 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885)
I20260926 04:08:02.295989 7534 ts_manager.cc:194] Registered new tserver with Master: 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885)
W20260926 04:08:02.379009 8059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:02.379283 8059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:02.379330 8059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:02.380817 8059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:02.380904 8059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:02.380955 8059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:02.382676 8059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:37477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=36281
--tserver_master_addrs=127.0.213.254:46751,127.0.213.253:45441,127.0.213.252:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:02.383044 8059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:02.383469 8059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:02.389464 8065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.389430 8064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:02.389441 8067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:02.389650 8059 server_base.cc:1034] running on GCE node
I20260926 04:08:02.389900 8059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:02.390175 8059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:02.391389 8059 hybrid_clock.cc:648] HybridClock initialized: now 1790395682391353 us; error 55 us; skew 500 ppm
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:02.394264 8059 init.cc:377] Logged in from keytab as oryx/127.0.213.195@KRBTEST.COM (short username oryx)
I20260926 04:08:02.394781 8059 webserver.cc:460] Webserver started at http://127.0.213.195:36281/ using document root <none> and password file <none>
I20260926 04:08:02.395038 8059 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:02.395098 8059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:02.396934 8059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:08:02.397670 8074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.397962 8059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:02.398061 8059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "cb6382895bb340558b73b8549a65e4ec"
format_stamp: "Formatted at 2026-09-26 04:08:01 on dist-test-slave-xf24"
server_key: "6d8eb7bc1403dd747462417a39296f71"
server_key_iv: "8e5d2b516145ed0290ba66b5d9cfa6ed"
server_key_version: "encryptionkey@0"
I20260926 04:08:02.398491 8059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:02.402000 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.398717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.0.1:50950)
0926 04:08:02.398786 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.398800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.398897 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.399151 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.399165 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.399168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:02.399259 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.399261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.400188 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.400189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.400918 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.400920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.401776 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.401777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.401833 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.401837 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.401896 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.401941 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 04:08:02.402022 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.398799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.0.213.252:43985)
0926 04:08:02.398986 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.398988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.398996 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:02.399022 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.399023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.399046 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.399115 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:02.399347 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.400105 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.401014 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.401648 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.401738 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.401814 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.401913 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.401953 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260926 04:08:02.402382 7746 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 04:08:02.402513 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6022f2cb9cc14142b3e639b0c99783ca. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.402637 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: LEARNER
I20260926 04:08:02.418398 8059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:02.420754 8059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:02.420904 8059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:02.421175 8059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:02.421509 8059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:02.421563 8059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.421600 8059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:02.421629 8059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:02.428311 8059 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:37477
I20260926 04:08:02.428351 8187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:37477 every 8 connection(s)
I20260926 04:08:02.428778 8059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:08:02.429940 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 8059
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.253@KRBTEST.COM
I20260926 04:08:02.440348 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.431052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.213.252:43985)
0926 04:08:02.431479 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.431482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.431644 (+ 162us) server_negotiation.cc:407] Connection header received
0926 04:08:02.431672 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.431673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.431685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.431745 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.432617 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.433392 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.434434 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.434739 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.434796 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.436721 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.436723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.436723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.436731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.438887 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.439320 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.439321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.439322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.439380 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.439584 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.439585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.439585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.439725 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.439837 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.440125 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.440206 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":406,"thread_start_us":66,"threads_started":1}
Sep 26 04:08:02 dist-test-slave-xf24 krb5kdc[6899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395682, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.252@KRBTEST.COM
I20260926 04:08:02.442312 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.429708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.213.195:48475)
0926 04:08:02.430289 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.430298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.431419 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.431947 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.431948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.432455 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.432773 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.432780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.433676 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.433677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.435020 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.435022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.435660 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.435661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.435752 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.437103 (+ 1351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.437108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.438729 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.440607 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.440616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.440618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.440941 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.441145 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.441146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.441146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.441287 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.441584 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.441590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.441785 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.441901 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.442073 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":75,"thread_start_us":85,"threads_started":1}
I20260926 04:08:02.442723 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.430015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:60491)
0926 04:08:02.430431 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.430439 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.431563 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.431870 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.431874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.432453 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.432773 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.432780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.433684 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.433685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.435023 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.435025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.435522 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.435523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.435679 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.436342 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.436353 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.439107 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.441630 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.441630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.441631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.441836 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.442048 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.442049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.442049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.442102 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.442407 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.442408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.442457 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.442521 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.442615 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":48,"thread_start_us":75,"threads_started":1}
I20260926 04:08:02.442775 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.429700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:48475 (local address 127.0.213.253:45441)
0926 04:08:02.429752 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.429754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.430378 (+ 624us) server_negotiation.cc:407] Connection header received
0926 04:08:02.431549 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.431550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.431561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.431616 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.432880 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.433574 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.435211 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.435535 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.435605 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.438848 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.438849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.438849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.438857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.440517 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.441026 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.441027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.441027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.441084 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.441363 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.441365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.441365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.441501 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.441557 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.442580 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.442669 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:08:02.443104 8190 heartbeater.cc:347] Connected to a master server at 127.0.213.253:45441
I20260926 04:08:02.443200 8190 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.443914 8190 heartbeater.cc:511] Master 127.0.213.253:45441 requested a full tablet report, sending...
I20260926 04:08:02.444133 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.430129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:60491 (local address 127.0.213.254:46751)
0926 04:08:02.430198 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.430199 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.430500 (+ 301us) server_negotiation.cc:407] Connection header received
0926 04:08:02.431667 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.431668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.431678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.431727 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.432881 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.433569 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.435113 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.435400 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.435458 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.439227 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.439229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.439229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.439237 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.441532 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.441924 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.441925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.441926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.441981 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.442170 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.442171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.442171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.442316 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.442380 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.442908 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.442992 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:02.444536 7534 ts_manager.cc:194] Registered new tserver with Master: cb6382895bb340558b73b8549a65e4ec (127.0.213.195:37477)
I20260926 04:08:02.444840 8189 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46751
I20260926 04:08:02.444885 8189 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.444978 8189 heartbeater.cc:511] Master 127.0.213.254:46751 requested a full tablet report, sending...
I20260926 04:08:02.445385 6944 ts_manager.cc:194] Re-registered known tserver with Master: cb6382895bb340558b73b8549a65e4ec (127.0.213.195:37477)
I20260926 04:08:02.445549 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.431004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.0.213.254:46751)
0926 04:08:02.431155 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.431157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.431342 (+ 185us) server_negotiation.cc:407] Connection header received
0926 04:08:02.431373 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.431373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.431386 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.431444 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.432362 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.433202 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.434203 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.434559 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.434621 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.436968 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.436969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.436970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.437229 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.439390 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.442365 (+ 2975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.442366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.442367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.443318 (+ 951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.443602 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.443603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.443603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.443777 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.443882 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.445290 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.445382 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260926 04:08:02.445887 6944 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:60491
I20260926 04:08:02.447496 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.429704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:43721 (local address 127.0.213.252:43985)
0926 04:08:02.429778 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.429780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.430046 (+ 266us) server_negotiation.cc:407] Connection header received
0926 04:08:02.431695 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.431696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.431711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.431751 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.436194 (+ 4443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.436984 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.439692 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.440025 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.440097 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.443183 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.443184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.443185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.443192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.445084 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.446592 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.446593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.446593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.446654 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.446896 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.446897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.446898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.447037 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.447081 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.447288 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.447377 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":55}
I20260926 04:08:02.448177 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.431436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.213.253:45441)
0926 04:08:02.435089 (+ 3653us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.435091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.435099 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:02.435120 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.435121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.435134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.437975 (+ 2841us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.438017 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.441152 (+ 3135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.442685 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.443323 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.443634 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.445028 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.445030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.445030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.445039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.446928 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.447334 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.447335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.447335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.447390 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.447617 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.447618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.447619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.447764 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.447862 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.447979 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.448082 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3629,"thread_start_us":61,"threads_started":1}
I20260926 04:08:02.448221 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.429708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.213.195:43721)
0926 04:08:02.429887 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.429938 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.431621 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.431869 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.431874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.436002 (+ 4128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:02.436113 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.436115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.438696 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.438698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.439582 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.439584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.440180 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.440181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.440256 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.440925 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:02.440930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:02.443053 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:02.445173 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.445173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.445174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.445682 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.446755 (+ 1073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:02.446756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:02.446756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:02.446819 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:02.447128 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:02.447129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:02.447170 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.447655 (+ 485us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.447780 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260926 04:08:02.448725 8188 heartbeater.cc:347] Connected to a master server at 127.0.213.252:43985
I20260926 04:08:02.448786 8188 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:02.448872 8188 heartbeater.cc:511] Master 127.0.213.252:43985 requested a full tablet report, sending...
I20260926 04:08:02.449183 7736 ts_manager.cc:194] Registered new tserver with Master: cb6382895bb340558b73b8549a65e4ec (127.0.213.195:37477)
I20260926 04:08:02.447862 6943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:08:02.450059 6943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:02.450810 7690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: NON_VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: true } } }
I20260926 04:08:02.450922 7690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:02.452121 7504 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: attempting to promote NON_VOTER 19ff866ecffc4ff096f4f72eab95001f to VOTER
I20260926 04:08:02.452332 7504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } }
I20260926 04:08:02.452726 7552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Refusing update from remote peer 6022f2cb9cc14142b3e639b0c99783ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 04:08:02.453222 7700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 04:08:02.453321 7746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 LEARNER]: Refusing update from remote peer 6022f2cb9cc14142b3e639b0c99783ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 04:08:02.453691 7700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 04:08:02.453826 7504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.453992 7552 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.454113 7746 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.454880 7504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.454965 7504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:02.454945 7696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.455027 7696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:08:02.455019 7504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.455076 7504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:02.455031 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.455114 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:08:02.455477 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6022f2cb9cc14142b3e639b0c99783ca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:02.455556 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:02.460954 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.456243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41619 (local address 127.0.0.1:56154)
0926 04:08:02.456303 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.456316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.456365 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.457266 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.457268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.457270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:02.457321 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.457323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.458616 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.458618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.459696 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.459699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.460700 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.460701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.460761 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.460765 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.460833 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.460874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":26}
I20260926 04:08:02.461028 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.456431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.0.213.193:41619)
0926 04:08:02.456636 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.456641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.456653 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:08:02.457144 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.457146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.457162 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.457257 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:02.457384 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.458505 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.459799 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.460608 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.460684 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.460772 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.460849 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.460940 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":128,"threads_started":1}
I20260926 04:08:02.461413 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.456349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:37477 (local address 127.0.0.1:53336)
0926 04:08:02.456544 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.456553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.456596 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.457298 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.457299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.457302 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:02.457362 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.457364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.458625 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.458626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.459974 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.459977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.461120 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.461121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.461194 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.461198 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.461270 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.461317 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":47,"thread_start_us":64,"threads_started":1}
I20260926 04:08:02.461620 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.456550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.0.213.195:37477)
0926 04:08:02.456704 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.456708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.456716 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:02.457143 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.457145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.457163 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.457435 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:02.457490 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.458531 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.460111 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.460909 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.460989 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.461057 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.461488 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.461539 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 04:08:02.461947 7849 tablet_service.cc:1467] Processing CreateTablet for tablet f335a7f0bfbc4e8ba67c0cde49705e5b (DEFAULT_TABLE table=test-table [id=9cc2270b83e148cda176546df4e8a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:02.462350 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f335a7f0bfbc4e8ba67c0cde49705e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:02.462272 8122 tablet_service.cc:1467] Processing CreateTablet for tablet f335a7f0bfbc4e8ba67c0cde49705e5b (DEFAULT_TABLE table=test-table [id=9cc2270b83e148cda176546df4e8a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:02.462450 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.456243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45885 (local address 127.0.0.1:43508)
0926 04:08:02.456415 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.456426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.456474 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.458900 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.458901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.458903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:02.458939 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.458940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.460196 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.460197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.461074 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.461076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.462180 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.462181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.462255 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.462259 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.462323 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.462382 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":25,"thread_start_us":85,"threads_started":1}
I20260926 04:08:02.462483 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.456653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.0.213.194:45885)
0926 04:08:02.458699 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.458702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.458711 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:02.458788 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.458790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.458808 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.458891 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:02.459004 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.460135 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.461284 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.462063 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.462146 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.462226 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.462348 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.462394 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":88,"threads_started":1}
I20260926 04:08:02.462603 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f335a7f0bfbc4e8ba67c0cde49705e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:02.463058 7986 tablet_service.cc:1467] Processing CreateTablet for tablet f335a7f0bfbc4e8ba67c0cde49705e5b (DEFAULT_TABLE table=test-table [id=9cc2270b83e148cda176546df4e8a2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:02.463373 7986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f335a7f0bfbc4e8ba67c0cde49705e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:02.466426 8218 tablet_bootstrap.cc:492] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Bootstrap starting.
I20260926 04:08:02.466840 8219 tablet_bootstrap.cc:492] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Bootstrap starting.
I20260926 04:08:02.467051 8218 tablet_bootstrap.cc:654] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:02.467129 8217 tablet_bootstrap.cc:492] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Bootstrap starting.
I20260926 04:08:02.467381 8218 log.cc:824] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:02.467603 8219 tablet_bootstrap.cc:654] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:02.467736 8217 tablet_bootstrap.cc:654] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:02.468035 8217 log.cc:824] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:02.468025 8219 log.cc:824] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:02.468168 8218 tablet_bootstrap.cc:492] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: No bootstrap required, opened a new log
I20260926 04:08:02.468243 8218 ts_tablet_manager.cc:1397] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:08:02.468758 8219 tablet_bootstrap.cc:492] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: No bootstrap required, opened a new log
I20260926 04:08:02.468811 8219 ts_tablet_manager.cc:1397] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:08:02.470146 8218 raft_consensus.cc:348] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.470324 8218 raft_consensus.cc:374] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:02.470366 8218 raft_consensus.cc:729] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5152c40fae1409c96230d6c1d9d53a5, State: Initialized, Role: FOLLOWER
I20260926 04:08:02.470485 8218 consensus_queue.cc:260] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.470664 8219 raft_consensus.cc:348] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.470775 8218 ts_tablet_manager.cc:1428] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:08:02.470814 8219 raft_consensus.cc:374] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:02.470886 8219 raft_consensus.cc:729] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69955364d6cb442087ab54c46c4a016a, State: Initialized, Role: FOLLOWER
I20260926 04:08:02.470960 7916 heartbeater.cc:503] Master 127.0.213.254:46751 was elected leader, sending a full tablet report...
I20260926 04:08:02.471001 8219 consensus_queue.cc:260] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.471300 8219 ts_tablet_manager.cc:1428] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:08:02.471839 8053 heartbeater.cc:503] Master 127.0.213.254:46751 was elected leader, sending a full tablet report...
I20260926 04:08:02.472836 8217 tablet_bootstrap.cc:492] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: No bootstrap required, opened a new log
I20260926 04:08:02.473037 8217 ts_tablet_manager.cc:1397] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 04:08:02.475112 8217 raft_consensus.cc:348] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.475283 8217 raft_consensus.cc:374] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:02.475307 8217 raft_consensus.cc:729] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6382895bb340558b73b8549a65e4ec, State: Initialized, Role: FOLLOWER
I20260926 04:08:02.475405 8217 consensus_queue.cc:260] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.475715 8217 ts_tablet_manager.cc:1428] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:08:02.475850 8189 heartbeater.cc:503] Master 127.0.213.254:46751 was elected leader, sending a full tablet report...
W20260926 04:08:02.531096 8055 tablet.cc:2369] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:08:02.635486 7918 tablet.cc:2369] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:08:02.679791 8191 tablet.cc:2369] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:08:02.837474 8225 raft_consensus.cc:482] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:02.837604 8225 raft_consensus.cc:504] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.838056 8225 leader_election.cc:288] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619), 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885)
I20260926 04:08:02.841039 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.838389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:45433 (local address 127.0.213.194:45885)
0926 04:08:02.838448 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.838451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.838463 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:08:02.838523 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.838524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.838535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.838616 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:02.838811 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.839319 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.840299 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.840669 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.840731 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.840738 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.840920 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.840968 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:08:02.841045 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.838309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45885 (local address 127.0.213.195:45433)
0926 04:08:02.838400 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.838411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.838457 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.838640 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.838640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.838642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:02.838703 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.838704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.839422 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.839423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.840201 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.840203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.840754 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.840755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.840834 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.840840 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.840891 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.840923 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 04:08:02.841092 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.838395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:60531 (local address 127.0.213.193:41619)
0926 04:08:02.838456 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.838459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.838469 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:02.838500 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.838501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.838511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.838569 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:02.838757 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.839266 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.840355 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.840694 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.840788 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.840796 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.840980 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.841017 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 04:08:02.841159 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.838279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41619 (local address 127.0.213.195:60531)
0926 04:08:02.838375 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:02.838389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:02.838442 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:02.838595 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:02.838596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:02.838598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:02.838674 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.838676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.839355 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.839357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.840237 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:02.840239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.840808 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.840809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:02.840899 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.840903 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:02.840960 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:08:02.840997 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 04:08:02.841423 8006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f335a7f0bfbc4e8ba67c0cde49705e5b" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69955364d6cb442087ab54c46c4a016a" is_pre_election: true
I20260926 04:08:02.841437 7869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f335a7f0bfbc4e8ba67c0cde49705e5b" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5152c40fae1409c96230d6c1d9d53a5" is_pre_election: true
I20260926 04:08:02.841586 7869 raft_consensus.cc:2454] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 0.
I20260926 04:08:02.841586 8006 raft_consensus.cc:2454] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 0.
I20260926 04:08:02.841810 8075 leader_election.cc:302] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69955364d6cb442087ab54c46c4a016a, cb6382895bb340558b73b8549a65e4ec; no voters:
I20260926 04:08:02.841953 8225 raft_consensus.cc:2781] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:02.842010 8225 raft_consensus.cc:482] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:02.842034 8225 raft_consensus.cc:3037] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:02.842730 8225 raft_consensus.cc:504] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.842878 8225 leader_election.cc:288] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 election: Requested vote from peers c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619), 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885)
I20260926 04:08:02.843014 7869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f335a7f0bfbc4e8ba67c0cde49705e5b" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5152c40fae1409c96230d6c1d9d53a5"
I20260926 04:08:02.843088 7869 raft_consensus.cc:3037] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:02.843788 7869 raft_consensus.cc:2454] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 1.
I20260926 04:08:02.843884 8006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f335a7f0bfbc4e8ba67c0cde49705e5b" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69955364d6cb442087ab54c46c4a016a"
I20260926 04:08:02.843941 8006 raft_consensus.cc:3037] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:02.844072 8076 leader_election.cc:302] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5152c40fae1409c96230d6c1d9d53a5, cb6382895bb340558b73b8549a65e4ec; no voters:
I20260926 04:08:02.844206 8225 raft_consensus.cc:2781] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:02.844357 8225 raft_consensus.cc:686] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 1 LEADER]: Becoming Leader. State: Replica: cb6382895bb340558b73b8549a65e4ec, State: Running, Role: LEADER
I20260926 04:08:02.844643 8225 consensus_queue.cc:237] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } }
I20260926 04:08:02.844869 8006 raft_consensus.cc:2454] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 1.
I20260926 04:08:02.845623 6943 catalog_manager.cc:5495] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6382895bb340558b73b8549a65e4ec (127.0.213.195). New cstate: current_term: 1 leader_uuid: "cb6382895bb340558b73b8549a65e4ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } health_report { overall_health: HEALTHY } } }
I20260926 04:08:02.901870 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.899397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.0.213.195:37477)
0926 04:08:02.899475 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.899477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.899491 (+ 14us) server_negotiation.cc:407] Connection header received
0926 04:08:02.899527 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.899528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.899545 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.899624 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:02.899772 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.900308 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.901066 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.901263 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.901344 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.901517 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:02.901628 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:02.901727 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.901753 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.901774 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 04:08:02.903968 8006 raft_consensus.cc:1261] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Refusing update from remote peer cb6382895bb340558b73b8549a65e4ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:02.903968 7869 raft_consensus.cc:1261] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 1 FOLLOWER]: Refusing update from remote peer cb6382895bb340558b73b8549a65e4ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:02.904384 8225 consensus_queue.cc:1035] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:02.904543 8226 consensus_queue.cc:1035] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:02.907305 8229 mvcc.cc:205] Tried to move back new op lower bound from 7333460717171552256 to 7333460716932493312. Current Snapshot: MvccSnapshot[applied={T|T < 7333460717171552256}]
I20260926 04:08:02.908416 8231 mvcc.cc:205] Tried to move back new op lower bound from 7333460717171552256 to 7333460716932493312. Current Snapshot: MvccSnapshot[applied={T|T < 7333460717171552256}]
I20260926 04:08:02.911376 8230 mvcc.cc:205] Tried to move back new op lower bound from 7333460717171552256 to 7333460716932493312. Current Snapshot: MvccSnapshot[applied={T|T < 7333460717171552256 or (T in {7333460717171552256})}]
I20260926 04:08:02.911738 6943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:02.911842 6943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:02.913333 6943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6022f2cb9cc14142b3e639b0c99783ca: Sending DeleteTablet for 3 replicas of tablet f335a7f0bfbc4e8ba67c0cde49705e5b
I20260926 04:08:02.913653 7849 tablet_service.cc:1514] Processing DeleteTablet for tablet f335a7f0bfbc4e8ba67c0cde49705e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:02 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:56154
I20260926 04:08:02.913668 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet f335a7f0bfbc4e8ba67c0cde49705e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:02 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:53336
I20260926 04:08:02.913942 8239 tablet_replica.cc:330] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: stopping tablet replica
I20260926 04:08:02.914088 8239 raft_consensus.cc:2229] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:02.914182 7986 tablet_service.cc:1514] Processing DeleteTablet for tablet f335a7f0bfbc4e8ba67c0cde49705e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:02 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:43508
I20260926 04:08:02.914527 8238 tablet_replica.cc:330] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: stopping tablet replica
I20260926 04:08:02.914625 8238 raft_consensus.cc:2229] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:02.914752 8238 raft_consensus.cc:2258] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:02.914824 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 6912
I20260926 04:08:02.914837 8239 raft_consensus.cc:2258] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:02.916145 8239 ts_tablet_manager.cc:1905] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:02.916558 8238 ts_tablet_manager.cc:1905] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:02.918443 8239 ts_tablet_manager.cc:1918] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:02.918593 8239 log.cc:1196] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/f335a7f0bfbc4e8ba67c0cde49705e5b
I20260926 04:08:02.918622 8238 ts_tablet_manager.cc:1918] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:02.918685 8238 log.cc:1196] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/f335a7f0bfbc4e8ba67c0cde49705e5b
I20260926 04:08:02.918975 8238 ts_tablet_manager.cc:1939] T f335a7f0bfbc4e8ba67c0cde49705e5b P c5152c40fae1409c96230d6c1d9d53a5: Deleting consensus metadata
I20260926 04:08:02.918982 8239 ts_tablet_manager.cc:1939] T f335a7f0bfbc4e8ba67c0cde49705e5b P cb6382895bb340558b73b8549a65e4ec: Deleting consensus metadata
I20260926 04:08:02.914911 8240 tablet_replica.cc:330] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: stopping tablet replica
I20260926 04:08:02.919457 8240 raft_consensus.cc:2229] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:02.919606 8240 raft_consensus.cc:2258] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 04:08:02.921257 8077 connection.cc:535] server connection from 127.0.0.1:53336 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 04:08:02.921589 7802 connection.cc:535] server connection from 127.0.0.1:56154 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:08:02.927932 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.914372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.213.252:43985)
0926 04:08:02.914413 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.914416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.919743 (+ 5327us) server_negotiation.cc:407] Connection header received
0926 04:08:02.920180 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.920181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.920197 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.920240 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.921004 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.921673 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.922716 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.922984 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.923026 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.925022 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.925023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.925023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.925031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.926679 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.927087 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.927088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.927089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.927145 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.927350 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.927350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.927351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.927481 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.927557 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.927756 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.927841 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:08:02.929003 8240 ts_tablet_manager.cc:1905] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:02.940469 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.931642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.213.253:45441)
0926 04:08:02.931715 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.931717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.932001 (+ 284us) server_negotiation.cc:407] Connection header received
0926 04:08:02.932100 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.932102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.932121 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.932211 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.933254 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.934627 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.935609 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.935840 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.935932 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.937384 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.937385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.937386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.937393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.939148 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.939539 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.939541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.939541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.939603 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.939848 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.939849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.939850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.939978 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.940114 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.940243 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.940339 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:08:02.940640 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:02.932174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.213.252:43985)
0926 04:08:02.932220 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:02.932221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:02.932238 (+ 17us) server_negotiation.cc:407] Connection header received
0926 04:08:02.932259 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:02.932260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:02.932270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:02.932322 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:02.933225 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.933733 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.934499 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:02.934744 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:02.934811 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:02.937062 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:02.937065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:02.937066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:02.937074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:02.938851 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.939513 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.939514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.939515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.939603 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:02.939846 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:02.939847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:02.939847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:02.940111 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:02.940205 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:02.940449 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 04:08:02.940540 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:08:02.970856 8240 ts_tablet_manager.cc:1918] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:02.970989 8240 log.cc:1196] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/f335a7f0bfbc4e8ba67c0cde49705e5b
I20260926 04:08:02.971323 8240 ts_tablet_manager.cc:1939] T f335a7f0bfbc4e8ba67c0cde49705e5b P 69955364d6cb442087ab54c46c4a016a: Deleting consensus metadata
W20260926 04:08:02.981185 7939 connection.cc:441] server connection from 127.0.0.1:43508 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43508 (request call id 1) could send its response
I20260926 04:08:03.909533 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:03.909196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:45825)
0926 04:08:03.909399 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:03.909482 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260926 04:08:03.909533 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:03.909177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:60733)
0926 04:08:03.909409 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:03.909483 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260926 04:08:03.909773 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:03.909468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:45059)
0926 04:08:03.909680 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:03.909736 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":46,"threads_started":1}
W20260926 04:08:03.909816 7916 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:46751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
W20260926 04:08:03.910048 8053 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:46751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
W20260926 04:08:03.910072 8189 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:46751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
I20260926 04:08:04.413820 8252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6022f2cb9cc14142b3e639b0c99783ca)
I20260926 04:08:04.414004 8252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } }
I20260926 04:08:04.414239 8252 leader_election.cc:288] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751), 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253:45441)
I20260926 04:08:04.414605 8254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6022f2cb9cc14142b3e639b0c99783ca)
I20260926 04:08:04.414670 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.414422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50120)
0926 04:08:04.414564 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.414619 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
W20260926 04:08:04.414865 7722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
I20260926 04:08:04.414885 8254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } }
I20260926 04:08:04.415266 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.415175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50134)
0926 04:08:04.415214 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.415239 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 04:08:04.415376 7722 leader_election.cc:334] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751): Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
I20260926 04:08:04.415510 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.415265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50146)
0926 04:08:04.415412 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.415468 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":56,"threads_started":1}
I20260926 04:08:04.415586 8254 leader_election.cc:288] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751), 19ff866ecffc4ff096f4f72eab95001f (127.0.213.252:43985)
W20260926 04:08:04.415624 7528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
I20260926 04:08:04.416093 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.415967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50152)
0926 04:08:04.416022 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.416052 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:08:04.416180 7528 leader_election.cc:334] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751): Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
I20260926 04:08:04.417944 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.414640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:45441 (local address 127.0.0.1:52590)
0926 04:08:04.414737 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.414749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:04.414824 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:04.415257 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:04.415259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:04.415261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:04.415322 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.415326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.416328 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.416329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.417100 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.417101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.417672 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.417674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.417741 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.417745 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:04.417818 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:08:04.417866 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
I20260926 04:08:04.417940 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.414747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.0.213.253:45441)
0926 04:08:04.415078 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:04.415080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:04.415091 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:08:04.415123 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:04.415124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:04.415137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:04.415202 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:04.415408 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.416236 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.417191 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.417578 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.417670 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.417682 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:04.417825 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 04:08:04.417866 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":44,"threads_started":1}
I20260926 04:08:04.418381 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19ff866ecffc4ff096f4f72eab95001f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "24f67e30facb4c9a91d8672c0053ddd8" is_pre_election: true
I20260926 04:08:04.418490 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.415304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:43985 (local address 127.0.0.1:50990)
0926 04:08:04.415531 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.415542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:04.415704 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:04.415968 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:04.415969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:04.415971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:04.416018 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.416020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.416832 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.416833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.417605 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.417607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.418291 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.418292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.418357 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.418361 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:04.418415 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 04:08:04.418448 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":35,"threads_started":1}
I20260926 04:08:04.418525 7552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ff866ecffc4ff096f4f72eab95001f in term 1.
I20260926 04:08:04.418550 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.415490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.0.213.252:43985)
0926 04:08:04.415680 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:04.415682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:04.415693 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:08:04.415790 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:04.415791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:04.415802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:04.415907 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:04.416100 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.416732 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.417712 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.418213 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.418283 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.418291 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:04.418437 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 04:08:04.418481 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260926 04:08:04.418759 7722 leader_election.cc:302] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19ff866ecffc4ff096f4f72eab95001f, 24f67e30facb4c9a91d8672c0053ddd8; no voters: 6022f2cb9cc14142b3e639b0c99783ca
I20260926 04:08:04.418890 8252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 04:08:04.418949 8252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6022f2cb9cc14142b3e639b0c99783ca)
I20260926 04:08:04.418999 8252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:08:04.418959 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "24f67e30facb4c9a91d8672c0053ddd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19ff866ecffc4ff096f4f72eab95001f" is_pre_election: true
I20260926 04:08:04.419705 8252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } }
I20260926 04:08:04.419838 7746 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24f67e30facb4c9a91d8672c0053ddd8 in current term 2: Already voted for candidate 19ff866ecffc4ff096f4f72eab95001f in this term.
I20260926 04:08:04.419847 8252 leader_election.cc:288] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [CANDIDATE]: Term 2 election: Requested vote from peers 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751), 24f67e30facb4c9a91d8672c0053ddd8 (127.0.213.253:45441)
I20260926 04:08:04.420219 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.420011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50154)
0926 04:08:04.420130 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.420187 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 04:08:04.420192 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19ff866ecffc4ff096f4f72eab95001f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "24f67e30facb4c9a91d8672c0053ddd8"
I20260926 04:08:04.420202 7529 leader_election.cc:302] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24f67e30facb4c9a91d8672c0053ddd8; no voters: 19ff866ecffc4ff096f4f72eab95001f, 6022f2cb9cc14142b3e639b0c99783ca
I20260926 04:08:04.420276 7552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:08:04.420518 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.420428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50158)
0926 04:08:04.420473 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.420500 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 04:08:04.420598 7722 leader_election.cc:334] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751): Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
I20260926 04:08:04.421060 7552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ff866ecffc4ff096f4f72eab95001f in term 2.
I20260926 04:08:04.421177 8254 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:08:04.421286 7722 leader_election.cc:302] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19ff866ecffc4ff096f4f72eab95001f, 24f67e30facb4c9a91d8672c0053ddd8; no voters: 6022f2cb9cc14142b3e639b0c99783ca
I20260926 04:08:04.421383 8252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:08:04.421546 8252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [term 2 LEADER]: Becoming Leader. State: Replica: 19ff866ecffc4ff096f4f72eab95001f, State: Running, Role: LEADER
I20260926 04:08:04.421607 8252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } }
I20260926 04:08:04.422021 8262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19ff866ecffc4ff096f4f72eab95001f. Latest consensus state: current_term: 2 leader_uuid: "19ff866ecffc4ff096f4f72eab95001f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:04.422087 8262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:04.422298 8264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:04.422724 8264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:04.422822 8264 catalog_manager.cc:1259] Loaded cluster ID: 3f249b536a1740ce9f1292cd6e015d84
I20260926 04:08:04.422856 8264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:08:04.423019 8264 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:04.423112 8264 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Loaded TSK: 0
I20260926 04:08:04.423252 8264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:08:04.436421 7735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:08:04.438009 7735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:04.439127 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.439009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50180)
0926 04:08:04.439065 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.439101 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:08:04.439254 7552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [term 2 FOLLOWER]: Refusing update from remote peer 19ff866ecffc4ff096f4f72eab95001f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 04:08:04.439515 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.439415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.0.1:50188)
0926 04:08:04.439460 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.439487 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:08:04.439625 7722 consensus_peers.cc:487] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f -> Peer 6022f2cb9cc14142b3e639b0c99783ca (127.0.213.254:46751): Couldn't send request to peer 6022f2cb9cc14142b3e639b0c99783ca. Status: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 04:08:04.439782 8262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 04:08:04.440496 8254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19ff866ecffc4ff096f4f72eab95001f. Latest consensus state: current_term: 2 leader_uuid: "19ff866ecffc4ff096f4f72eab95001f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:04.440606 8254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:08:04.441013 8252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "19ff866ecffc4ff096f4f72eab95001f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:04.441099 8252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:04.441267 8254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "19ff866ecffc4ff096f4f72eab95001f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6022f2cb9cc14142b3e639b0c99783ca" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46751 } } peers { permanent_uuid: "24f67e30facb4c9a91d8672c0053ddd8" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 45441 } attrs { promote: false } } peers { permanent_uuid: "19ff866ecffc4ff096f4f72eab95001f" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 43985 } attrs { promote: false } } }
I20260926 04:08:04.441362 8254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f67e30facb4c9a91d8672c0053ddd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:08:04.446199 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.443182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.213.193:41619)
0926 04:08:04.443520 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:04.443522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:04.443529 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:04.443553 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:04.443554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:04.443565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:04.443772 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:04.443807 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.444501 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.445503 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.445874 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.445960 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.445972 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:04.446089 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 04:08:04.446131 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":50,"threads_started":1}
I20260926 04:08:04.446199 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.442975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41619 (local address 127.0.0.1:56158)
0926 04:08:04.443170 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.443179 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:04.443218 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:04.443659 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:04.443660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:04.443662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:04.443710 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.443712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.444580 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.444581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.445402 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.445405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.445960 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.445961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.446021 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.446025 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:04.446080 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:08:04.446121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260926 04:08:04.446352 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.442892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:37477 (local address 127.0.0.1:53348)
0926 04:08:04.443082 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.443091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:04.443130 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:04.443299 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:04.443301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:04.443302 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:04.443348 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.443349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.444363 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.444364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.445387 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.445388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.446136 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.446137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.446199 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.446203 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:04.446234 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 04:08:04.446280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 04:08:04.446360 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.442863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45885 (local address 127.0.0.1:43512)
0926 04:08:04.442904 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.442914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:04.443007 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:04.443450 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:04.443451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:04.443452 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:04.443483 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.443484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.444193 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.444194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.445340 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:04.445341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.446164 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.446165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:04.446216 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.446220 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:04.446264 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 04:08:04.446293 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 04:08:04.446527 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.442923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.0.213.194:45885)
0926 04:08:04.443309 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:04.443311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:04.443319 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:04.443345 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:04.443346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:04.443358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:04.443447 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:04.443563 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.444110 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.445443 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.446048 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.446126 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.446138 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:04.446386 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 04:08:04.446433 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":63,"threads_started":1}
I20260926 04:08:04.446728 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.443007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.0.213.195:37477)
0926 04:08:04.443143 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:04.443145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:04.443155 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:04.443184 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:04.443185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:04.443197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:04.443294 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:04.443471 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.444288 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.445471 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.446036 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.446577 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.446593 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:04.446614 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 04:08:04.446660 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260926 04:08:04.446722 7986 tablet_service.cc:1467] Processing CreateTablet for tablet 055d0994a3604cd6922e06588ef58fff (DEFAULT_TABLE table=test-table [id=8c770bde8ea34f8a911ece7b40391d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:04.446846 7986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055d0994a3604cd6922e06588ef58fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:04.446986 7849 tablet_service.cc:1467] Processing CreateTablet for tablet 055d0994a3604cd6922e06588ef58fff (DEFAULT_TABLE table=test-table [id=8c770bde8ea34f8a911ece7b40391d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:04.447090 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055d0994a3604cd6922e06588ef58fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:04.448204 8275 tablet_bootstrap.cc:492] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Bootstrap starting.
I20260926 04:08:04.448256 8122 tablet_service.cc:1467] Processing CreateTablet for tablet 055d0994a3604cd6922e06588ef58fff (DEFAULT_TABLE table=test-table [id=8c770bde8ea34f8a911ece7b40391d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:04.448375 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055d0994a3604cd6922e06588ef58fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:04.448596 8275 tablet_bootstrap.cc:654] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:04.449117 8275 tablet_bootstrap.cc:492] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: No bootstrap required, opened a new log
I20260926 04:08:04.449177 8275 ts_tablet_manager.cc:1397] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:04.449331 8275 raft_consensus.cc:348] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.449407 8275 raft_consensus.cc:374] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:04.449436 8275 raft_consensus.cc:729] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69955364d6cb442087ab54c46c4a016a, State: Initialized, Role: FOLLOWER
I20260926 04:08:04.449482 8275 consensus_queue.cc:260] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.449616 8275 ts_tablet_manager.cc:1428] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:04.450105 8279 tablet_bootstrap.cc:492] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Bootstrap starting.
I20260926 04:08:04.450546 8279 tablet_bootstrap.cc:654] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:04.450822 8276 tablet_bootstrap.cc:492] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5: Bootstrap starting.
I20260926 04:08:04.451175 8276 tablet_bootstrap.cc:654] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:04.451257 8279 tablet_bootstrap.cc:492] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: No bootstrap required, opened a new log
I20260926 04:08:04.451314 8279 ts_tablet_manager.cc:1397] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:04.451433 8279 raft_consensus.cc:348] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.451510 8279 raft_consensus.cc:374] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:04.451538 8279 raft_consensus.cc:729] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6382895bb340558b73b8549a65e4ec, State: Initialized, Role: FOLLOWER
I20260926 04:08:04.451586 8279 consensus_queue.cc:260] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.451607 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.449851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:56149)
0926 04:08:04.451509 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.451553 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1634,"thread_start_us":84,"threads_started":1}
I20260926 04:08:04.451725 8279 ts_tablet_manager.cc:1428] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:04.451979 8276 tablet_bootstrap.cc:492] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5: No bootstrap required, opened a new log
I20260926 04:08:04.452169 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.451863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:55921)
0926 04:08:04.452119 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.452148 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260926 04:08:04.452186 8188 heartbeater.cc:503] Master 127.0.213.252:43985 was elected leader, sending a full tablet report...
I20260926 04:08:04.452044 8276 ts_tablet_manager.cc:1397] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:04.452445 8276 raft_consensus.cc:348] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.452569 8276 raft_consensus.cc:374] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:04.452600 8276 raft_consensus.cc:729] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5152c40fae1409c96230d6c1d9d53a5, State: Initialized, Role: FOLLOWER
I20260926 04:08:04.452662 8276 consensus_queue.cc:260] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.452811 8276 ts_tablet_manager.cc:1428] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:04.452826 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.451942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:34403)
0926 04:08:04.452506 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.452541 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":73,"threads_started":1}
I20260926 04:08:04.453136 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.453053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:56657)
0926 04:08:04.453086 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.453114 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 04:08:04.453418 7915 heartbeater.cc:503] Master 127.0.213.252:43985 was elected leader, sending a full tablet report...
I20260926 04:08:04.453498 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.453136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:38635)
0926 04:08:04.453391 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.453425 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260926 04:08:04.453776 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.453701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:34463)
0926 04:08:04.453732 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.453756 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 04:08:04.476696 8283 raft_consensus.cc:482] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:04.476794 8283 raft_consensus.cc:504] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.476994 8283 leader_election.cc:288] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885), c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619)
I20260926 04:08:04.477276 8006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055d0994a3604cd6922e06588ef58fff" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69955364d6cb442087ab54c46c4a016a" is_pre_election: true
I20260926 04:08:04.477314 7869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055d0994a3604cd6922e06588ef58fff" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5152c40fae1409c96230d6c1d9d53a5" is_pre_election: true
I20260926 04:08:04.477392 8006 raft_consensus.cc:2454] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 0.
I20260926 04:08:04.477401 7869 raft_consensus.cc:2454] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 0.
I20260926 04:08:04.477578 8075 leader_election.cc:302] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69955364d6cb442087ab54c46c4a016a, cb6382895bb340558b73b8549a65e4ec; no voters:
I20260926 04:08:04.477728 8283 raft_consensus.cc:2781] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:04.477784 8283 raft_consensus.cc:482] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:04.477799 8283 raft_consensus.cc:3037] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:04.478590 8283 raft_consensus.cc:504] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.478760 8283 leader_election.cc:288] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 election: Requested vote from peers 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885), c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619)
I20260926 04:08:04.478883 8006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055d0994a3604cd6922e06588ef58fff" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69955364d6cb442087ab54c46c4a016a"
I20260926 04:08:04.478968 8006 raft_consensus.cc:3037] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:04.478915 7869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055d0994a3604cd6922e06588ef58fff" candidate_uuid: "cb6382895bb340558b73b8549a65e4ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5152c40fae1409c96230d6c1d9d53a5"
I20260926 04:08:04.479018 7869 raft_consensus.cc:3037] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:04.479532 8006 raft_consensus.cc:2454] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 1.
I20260926 04:08:04.479705 7869 raft_consensus.cc:2454] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb6382895bb340558b73b8549a65e4ec in term 1.
I20260926 04:08:04.479704 8075 leader_election.cc:302] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69955364d6cb442087ab54c46c4a016a, cb6382895bb340558b73b8549a65e4ec; no voters:
I20260926 04:08:04.479810 8283 raft_consensus.cc:2781] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:04.479975 8283 raft_consensus.cc:686] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 1 LEADER]: Becoming Leader. State: Replica: cb6382895bb340558b73b8549a65e4ec, State: Running, Role: LEADER
I20260926 04:08:04.480048 8283 consensus_queue.cc:237] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } }
I20260926 04:08:04.480487 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.480372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:37955)
0926 04:08:04.480410 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.480453 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 04:08:04.480600 7735 catalog_manager.cc:5495] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec reported cstate change: term changed from 0 to 1, leader changed from <none> to cb6382895bb340558b73b8549a65e4ec (127.0.213.195). New cstate: current_term: 1 leader_uuid: "cb6382895bb340558b73b8549a65e4ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6382895bb340558b73b8549a65e4ec" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 } health_report { overall_health: UNKNOWN } } }
I20260926 04:08:04.492712 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.490337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.213.195:37477)
0926 04:08:04.490404 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:04.490407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:04.490416 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:04.490479 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:04.490480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:04.490491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:04.490561 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:04.490716 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.491207 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.491949 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:04.492206 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:04.492280 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:04.492433 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:04.492467 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:04.492536 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:04.492598 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 04:08:04.492621 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 04:08:04.493724 8006 raft_consensus.cc:1261] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Refusing update from remote peer cb6382895bb340558b73b8549a65e4ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:04.493724 7869 raft_consensus.cc:1261] T 055d0994a3604cd6922e06588ef58fff P c5152c40fae1409c96230d6c1d9d53a5 [term 1 FOLLOWER]: Refusing update from remote peer cb6382895bb340558b73b8549a65e4ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:04.493960 8287 consensus_queue.cc:1035] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "69955364d6cb442087ab54c46c4a016a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:04.494109 8283 consensus_queue.cc:1035] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5152c40fae1409c96230d6c1d9d53a5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:04.494614 8052 heartbeater.cc:503] Master 127.0.213.252:43985 was elected leader, sending a full tablet report...
I20260926 04:08:04.495286 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.495176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:43489)
0926 04:08:04.495215 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.495250 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 04:08:04.495306 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.495028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:40797)
0926 04:08:04.495250 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.495276 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260926 04:08:04.496276 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.496112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:38585)
0926 04:08:04.496216 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.496246 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 04:08:04.496786 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.496681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.193:60147)
0926 04:08:04.496715 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.496748 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 04:08:04.496887 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.496741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.195:58973)
0926 04:08:04.496831 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.496857 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 04:08:04.497129 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.497024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46751 (local address 127.0.213.194:47807)
0926 04:08:04.497082 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.497105 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 04:08:04.498018 7735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:04.498122 7735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:04.499476 7735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19ff866ecffc4ff096f4f72eab95001f: Sending DeleteTablet for 3 replicas of tablet 055d0994a3604cd6922e06588ef58fff
I20260926 04:08:04.499756 7986 tablet_service.cc:1514] Processing DeleteTablet for tablet 055d0994a3604cd6922e06588ef58fff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:04 UTC) from {username='oryx', principal='oryx/127.0.213.252@KRBTEST.COM'} at 127.0.0.1:43512
I20260926 04:08:04.499797 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 055d0994a3604cd6922e06588ef58fff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:04 UTC) from {username='oryx', principal='oryx/127.0.213.252@KRBTEST.COM'} at 127.0.0.1:53348
I20260926 04:08:04.499970 7849 tablet_service.cc:1514] Processing DeleteTablet for tablet 055d0994a3604cd6922e06588ef58fff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:04 UTC) from {username='oryx', principal='oryx/127.0.213.252@KRBTEST.COM'} at 127.0.0.1:56158
I20260926 04:08:04.500077 8298 tablet_replica.cc:330] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: stopping tablet replica
I20260926 04:08:04.500159 8298 raft_consensus.cc:2229] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:04.500173 8299 tablet_replica.cc:330] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: stopping tablet replica
I20260926 04:08:04.500221 8298 raft_consensus.cc:2258] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:04.500227 8299 raft_consensus.cc:2229] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:04.500221 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7689
I20260926 04:08:04.501507 8298 ts_tablet_manager.cc:1905] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:04.501530 8299 raft_consensus.cc:2258] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:04.501865 8299 ts_tablet_manager.cc:1905] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:04.503247 8298 ts_tablet_manager.cc:1918] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:04.503302 8298 log.cc:1196] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/055d0994a3604cd6922e06588ef58fff
I20260926 04:08:04.503588 8298 ts_tablet_manager.cc:1939] T 055d0994a3604cd6922e06588ef58fff P 69955364d6cb442087ab54c46c4a016a: Deleting consensus metadata
I20260926 04:08:04.503717 8299 ts_tablet_manager.cc:1918] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:04.503782 8299 log.cc:1196] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/055d0994a3604cd6922e06588ef58fff
I20260926 04:08:04.503921 7720 catalog_manager.cc:4841] TS 69955364d6cb442087ab54c46c4a016a (127.0.213.194:45885): tablet 055d0994a3604cd6922e06588ef58fff (table test-table [id=8c770bde8ea34f8a911ece7b40391d4e]) successfully deleted
I20260926 04:08:04.504025 8299 ts_tablet_manager.cc:1939] T 055d0994a3604cd6922e06588ef58fff P cb6382895bb340558b73b8549a65e4ec: Deleting consensus metadata
I20260926 04:08:04.504365 7721 catalog_manager.cc:4841] TS cb6382895bb340558b73b8549a65e4ec (127.0.213.195:37477): tablet 055d0994a3604cd6922e06588ef58fff (table test-table [id=8c770bde8ea34f8a911ece7b40391d4e]) successfully deleted
I20260926 04:08:04.506690 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7923
I20260926 04:08:04.506731 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:04.506616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41619 (local address 127.0.0.1:56174)
0926 04:08:04.506662 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:04.506690 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:41619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 04:08:04.506937 7721 catalog_manager.cc:4584] TS c5152c40fae1409c96230d6c1d9d53a5 (127.0.213.193:41619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 055d0994a3604cd6922e06588ef58fff: Network error: Client connection negotiation failed: client connection to 127.0.213.193:41619: connect: Connection refused (error 111)
I20260926 04:08:04.512279 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 8059
I20260926 04:08:04.518632 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7408
I20260926 04:08:04.523757 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 7596
2026-09-26T04:08:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4932 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:08:04 dist-test-slave-xf24 krb5kdc[8303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:08:04 dist-test-slave-xf24 krb5kdc[8303](info): set up 2 sockets
Sep 26 04:08:04 dist-test-slave-xf24 krb5kdc[8303](info): commencing operation
krb5kdc: starting...
W20260926 04:08:05.624616 855 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:08:05 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab.
Entry for principal oryx/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab.
I20260926 04:08:05.777195 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41209
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab
--principal=oryx/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:41209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:05.864184 8316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:05.864416 8316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:05.864442 8316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:05.865684 8316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:05.865728 8316 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 04:08:05.865741 8316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:05.865754 8316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:05.865767 8316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:05.865779 8316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:05.867223 8316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:41209
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.254.keytab
--principal=oryx/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:05.867419 8316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:05.867600 8316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:05.872905 8322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:05.872910 8321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:05.873008 8324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:05.873289 8316 server_base.cc:1034] running on GCE node
I20260926 04:08:05.873528 8316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:05.873796 8316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:05.874958 8316 hybrid_clock.cc:648] HybridClock initialized: now 1790395685874944 us; error 32 us; skew 500 ppm
Sep 26 04:08:05 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395685, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:05.877763 8316 init.cc:377] Logged in from keytab as oryx/127.0.213.254@KRBTEST.COM (short username oryx)
I20260926 04:08:05.878283 8316 webserver.cc:460] Webserver started at http://127.0.213.254:41563/ using document root <none> and password file <none>
I20260926 04:08:05.878535 8316 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:05.878582 8316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:05.878738 8316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:05.880107 8316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "b12717eb25b74b7c93defae73697c20e"
format_stamp: "Formatted at 2026-09-26 04:08:05 on dist-test-slave-xf24"
server_key: "6e7e05cc9b526e97215896afa0cf847d"
server_key_iv: "453e7570146c860d241718909c453904"
server_key_version: "encryptionkey@0"
I20260926 04:08:05.880569 8316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "b12717eb25b74b7c93defae73697c20e"
format_stamp: "Formatted at 2026-09-26 04:08:05 on dist-test-slave-xf24"
server_key: "6e7e05cc9b526e97215896afa0cf847d"
server_key_iv: "453e7570146c860d241718909c453904"
server_key_version: "encryptionkey@0"
I20260926 04:08:05.882002 8316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:08:05.882712 8331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:05.883034 8316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:05.883142 8316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "b12717eb25b74b7c93defae73697c20e"
format_stamp: "Formatted at 2026-09-26 04:08:05 on dist-test-slave-xf24"
server_key: "6e7e05cc9b526e97215896afa0cf847d"
server_key_iv: "453e7570146c860d241718909c453904"
server_key_version: "encryptionkey@0"
I20260926 04:08:05.883237 8316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:05.898036 8316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:05.900370 8316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:05.900543 8316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:05.905689 8316 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:41209
I20260926 04:08:05.905735 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:41209 every 8 connection(s)
I20260926 04:08:05.906111 8316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:05.906756 8394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:05.909124 8394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Bootstrap starting.
I20260926 04:08:05.909739 8394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:05.910063 8394 log.cc:824] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:05.910828 8394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: No bootstrap required, opened a new log
I20260926 04:08:05.912963 8394 raft_consensus.cc:348] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12717eb25b74b7c93defae73697c20e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41209 } }
I20260926 04:08:05.913122 8394 raft_consensus.cc:374] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:05.913146 8394 raft_consensus.cc:729] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12717eb25b74b7c93defae73697c20e, State: Initialized, Role: FOLLOWER
I20260926 04:08:05.913307 8394 consensus_queue.cc:260] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12717eb25b74b7c93defae73697c20e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41209 } }
I20260926 04:08:05.913390 8394 raft_consensus.cc:388] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:05.913415 8394 raft_consensus.cc:482] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:05.913443 8394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:05.913637 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 8316
I20260926 04:08:05.913738 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:08:05.913921 855 external_mini_cluster.cc:1428] Setting key 44542fe6b17844bd0b72bc858ae5ae57
I20260926 04:08:05.914171 8394 raft_consensus.cc:504] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12717eb25b74b7c93defae73697c20e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41209 } }
I20260926 04:08:05.914296 8394 leader_election.cc:302] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b12717eb25b74b7c93defae73697c20e; no voters:
I20260926 04:08:05.914489 8394 leader_election.cc:288] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:08:05.914546 8398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:05.914804 8398 raft_consensus.cc:686] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [term 1 LEADER]: Becoming Leader. State: Replica: b12717eb25b74b7c93defae73697c20e, State: Running, Role: LEADER
I20260926 04:08:05.914882 8394 sys_catalog.cc:564] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:05.914963 8398 consensus_queue.cc:237] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12717eb25b74b7c93defae73697c20e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41209 } }
I20260926 04:08:05.915444 8400 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b12717eb25b74b7c93defae73697c20e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12717eb25b74b7c93defae73697c20e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41209 } } }
I20260926 04:08:05.915529 8400 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:05.916115 8401 sys_catalog.cc:455] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b12717eb25b74b7c93defae73697c20e. Latest consensus state: current_term: 1 leader_uuid: "b12717eb25b74b7c93defae73697c20e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b12717eb25b74b7c93defae73697c20e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41209 } } }
I20260926 04:08:05.916205 8401 sys_catalog.cc:458] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e [sys.catalog]: This master's current role is: LEADER
W20260926 04:08:05.916805 8415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:08:05.916868 8415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:08:05.916923 8408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:05.917128 8408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:05.918761 8408 catalog_manager.cc:1347] Generated new cluster ID: 0012162b0915499fac41b7d670b3dc58
I20260926 04:08:05.918823 8408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:08:05 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:05.926169 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:05.914522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.0.213.254:41209)
0926 04:08:05.914783 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:05.914794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:05.914837 (+ 43us) server_negotiation.cc:407] Connection header received
0926 04:08:05.914990 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:05.914998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:05.915069 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:05.915236 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:05.916109 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:05.917456 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:05.918277 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:05.918944 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:05.919067 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:05.922078 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:05.922083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:05.922083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:05.922098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:05.924359 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:05.924803 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:05.924807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:05.924807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:05.924949 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:05.925166 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:05.925167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:05.925167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:05.925515 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:05.925695 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:05.925771 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 04:08:05.925930 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
I20260926 04:08:05.928891 8408 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:08:05.929446 8408 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:05.934289 8408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Generated new TSK 0
I20260926 04:08:05.934551 8408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab.
Entry for principal oryx/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab.
I20260926 04:08:05.969030 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41209
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:06.055761 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:06.056020 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:06.056061 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:06.057323 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:06.057400 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:06.057439 8420 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:08:06.057499 8420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:06.059072 8420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:41209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:06.059353 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:06.059592 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:06.065512 8428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:06.065506 8425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:06.065544 8420 server_base.cc:1034] running on GCE node
W20260926 04:08:06.065506 8426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:06.066007 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:06.066287 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:06.067448 8420 hybrid_clock.cc:648] HybridClock initialized: now 1790395686067420 us; error 46 us; skew 500 ppm
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:06.070281 8420 init.cc:377] Logged in from keytab as oryx/127.0.213.193@KRBTEST.COM (short username oryx)
I20260926 04:08:06.071102 8420 webserver.cc:460] Webserver started at http://127.0.213.193:43869/ using document root <none> and password file <none>
I20260926 04:08:06.071381 8420 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:06.071455 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:06.071617 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:06.072785 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "1f833f6dbcf94eec981d0e950593b4b9"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "6ce10780f00736b448541fbd321950b5"
server_key_iv: "41f561c2ade34ced3119a77a62a0a541"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.073181 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "1f833f6dbcf94eec981d0e950593b4b9"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "6ce10780f00736b448541fbd321950b5"
server_key_iv: "41f561c2ade34ced3119a77a62a0a541"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.074617 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:06.075456 8435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.075639 8420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.075745 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "1f833f6dbcf94eec981d0e950593b4b9"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "6ce10780f00736b448541fbd321950b5"
server_key_iv: "41f561c2ade34ced3119a77a62a0a541"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.075821 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:06.086583 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:06.089473 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:06.089676 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:06.090606 8420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:06.090687 8420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.090900 8420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:06.090992 8420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:06.098881 8420 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:44941
I20260926 04:08:06.098975 8550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:44941 every 8 connection(s)
I20260926 04:08:06.099300 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:06.102504 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.091051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:56647 (local address 127.0.213.254:41209)
0926 04:08:06.091125 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.091127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.091375 (+ 248us) server_negotiation.cc:407] Connection header received
0926 04:08:06.092165 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.092166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.092179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.092238 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.093308 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.094029 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.095882 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.096278 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.096407 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.099330 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.099332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.099332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.099341 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.101004 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.101461 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.101462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.101462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.101523 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.101785 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.101786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.101787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.101916 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.101995 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.102275 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.102387 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 04:08:06.102674 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.091006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.213.193:56647)
0926 04:08:06.091180 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.091228 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.091992 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.092361 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.092366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.092841 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.093182 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.093191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.094167 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.094168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.095745 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.095749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.096430 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.096431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.096613 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.097220 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.097231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.099205 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.101114 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.101121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.101123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.101361 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.101603 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.101604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.101604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.101716 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.101984 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.101991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.102140 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.102257 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.102425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260926 04:08:06.103228 8551 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41209
I20260926 04:08:06.103318 8551 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:06.103510 8551 heartbeater.cc:511] Master 127.0.213.254:41209 requested a full tablet report, sending...
I20260926 04:08:06.104094 8348 ts_manager.cc:194] Registered new tserver with Master: 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941)
I20260926 04:08:06.104902 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 8420
I20260926 04:08:06.104890 8348 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:56647
I20260926 04:08:06.105006 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:08:06.105219 855 external_mini_cluster.cc:1428] Setting key 46cb2daada2d1c9e627e359718337a9f
I20260926 04:08:06.110937 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.104399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:54399 (local address 127.0.213.254:41209)
0926 04:08:06.104452 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.104455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.104561 (+ 106us) server_negotiation.cc:407] Connection header received
0926 04:08:06.104625 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.104626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.104637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.104694 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.105025 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.105621 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.106461 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.106671 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.106755 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.107884 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.107885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.107885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.107892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.109635 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.110102 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.110103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.110103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.110164 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.110415 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.110416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.110416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.110537 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.110602 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.110794 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.110869 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:08:06.111083 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.104342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.213.193:54399)
0926 04:08:06.104489 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.104499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.104576 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.104699 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.104699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.104886 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.104938 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.104940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.105777 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.105778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.106358 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.106360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.106759 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.106760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.106837 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.107319 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.107325 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.107768 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.109726 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.109727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.109728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.109999 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.110245 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.110246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.110246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.110325 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.110615 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.110617 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.110677 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.110771 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.110913 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab.
Entry for principal oryx/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab.
I20260926 04:08:06.137007 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41209
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:06.224112 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:06.224339 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:06.224365 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:06.225623 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:06.225677 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:06.225698 8560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:08:06.225718 8560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:06.227280 8560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:41209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:06.227550 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:06.227838 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:06.233157 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:06.233215 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:06.233157 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:06.233384 8560 server_base.cc:1034] running on GCE node
I20260926 04:08:06.233690 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:06.233948 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:06.235114 8560 hybrid_clock.cc:648] HybridClock initialized: now 1790395686235083 us; error 43 us; skew 500 ppm
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:06.237949 8560 init.cc:377] Logged in from keytab as oryx/127.0.213.194@KRBTEST.COM (short username oryx)
I20260926 04:08:06.238445 8560 webserver.cc:460] Webserver started at http://127.0.213.194:42361/ using document root <none> and password file <none>
I20260926 04:08:06.238705 8560 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:06.238777 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:06.238931 8560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:06.240013 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "fb912d2079654ebe8f0e77ca68f7c4cf"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "2f341c7b156f6f870fd42284ca2cf7de"
server_key_iv: "0c93b97e93c5ca6ab6a03805d0926ae7"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.240456 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "fb912d2079654ebe8f0e77ca68f7c4cf"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "2f341c7b156f6f870fd42284ca2cf7de"
server_key_iv: "0c93b97e93c5ca6ab6a03805d0926ae7"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.241846 8560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.242655 8575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.242866 8560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.242969 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "fb912d2079654ebe8f0e77ca68f7c4cf"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "2f341c7b156f6f870fd42284ca2cf7de"
server_key_iv: "0c93b97e93c5ca6ab6a03805d0926ae7"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.243052 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:06.268595 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:06.271257 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:06.271448 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:06.272095 8560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:06.272166 8560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.272225 8560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:06.272256 8560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:06.279814 8560 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:34569
I20260926 04:08:06.279968 8690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:34569 every 8 connection(s)
I20260926 04:08:06.280247 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:08:06.282888 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 8560
I20260926 04:08:06.283034 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:08:06.283231 855 external_mini_cluster.cc:1428] Setting key 051e36513f4545ad25fe08aee006ddf4
I20260926 04:08:06.283385 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.272514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:36211 (local address 127.0.213.254:41209)
0926 04:08:06.272587 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.272589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.272838 (+ 249us) server_negotiation.cc:407] Connection header received
0926 04:08:06.273350 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.273352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.273366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.273454 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.274237 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.275073 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.276381 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.276604 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.276694 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.280360 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.280362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.280363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.280371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.281893 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.282327 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.282329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.282329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.282407 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.282684 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.282684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.282685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.282817 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.282891 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.283203 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.283286 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:06.283617 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.272525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.213.194:36211)
0926 04:08:06.272696 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.272739 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.273251 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.273509 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.273512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.273761 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.273959 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.273963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.275187 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.275189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.276271 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.276275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.276705 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.276706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.276837 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.277436 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.277448 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.280219 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.281997 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.282005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.282007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.282241 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.282475 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.282475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.282476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.282604 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.282882 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.282888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.283068 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.283182 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.283362 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260926 04:08:06.284253 8691 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41209
I20260926 04:08:06.284344 8691 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:06.284578 8691 heartbeater.cc:511] Master 127.0.213.254:41209 requested a full tablet report, sending...
I20260926 04:08:06.284973 8347 ts_manager.cc:194] Registered new tserver with Master: fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:06.285506 8347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:36211
WARNING: no policy specified for oryx/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:08:06.292629 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.285878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:58031 (local address 127.0.213.254:41209)
0926 04:08:06.285917 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.285919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.285981 (+ 62us) server_negotiation.cc:407] Connection header received
0926 04:08:06.286081 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.286082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.286093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.286123 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.286592 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.287470 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.288271 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.288591 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.288662 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.289758 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.289759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.289759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.289765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.291295 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.291710 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.291711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.291711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.291764 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.292102 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.292103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.292103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.292252 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.292315 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.292498 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.292563 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:08:06.292714 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.285736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.213.194:58031)
0926 04:08:06.285908 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.285920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.286009 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.286186 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.286187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.286420 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.286488 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.286490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.287578 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.287579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.288193 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.288195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.288669 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.288670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.288736 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.289206 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.289212 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.289662 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.291385 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.291386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.291387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.291627 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.291884 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.291885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.291885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.291992 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.292328 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.292329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.292389 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.292469 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.292585 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab.
Entry for principal oryx/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab.
I20260926 04:08:06.312457 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41209
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:06.397533 8700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:06.397795 8700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:06.397840 8700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:06.399128 8700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:06.399205 8700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:06.399241 8700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 04:08:06.399290 8700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:06.401031 8700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.213.254:41209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:06.401336 8700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:06.401656 8700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:06.407109 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:06.407204 8708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:06.407388 8700 server_base.cc:1034] running on GCE node
W20260926 04:08:06.407120 8706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:06.407691 8700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:06.407944 8700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:06.409123 8700 hybrid_clock.cc:648] HybridClock initialized: now 1790395686409125 us; error 48 us; skew 500 ppm
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:06.411808 8700 init.cc:377] Logged in from keytab as oryx/127.0.213.195@KRBTEST.COM (short username oryx)
I20260926 04:08:06.412360 8700 webserver.cc:460] Webserver started at http://127.0.213.195:33281/ using document root <none> and password file <none>
I20260926 04:08:06.412629 8700 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:06.412698 8700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:06.412851 8700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:06.413944 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "d057124d1d6e4bb1bebb81000776b0cd"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "9119bee25e5c621bcc56ec1afdec054c"
server_key_iv: "0f408f79e3e91c2c8fc5f84e5b71464c"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.414307 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "d057124d1d6e4bb1bebb81000776b0cd"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "9119bee25e5c621bcc56ec1afdec054c"
server_key_iv: "0f408f79e3e91c2c8fc5f84e5b71464c"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.415923 8700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:06.416965 8715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.417176 8700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.417308 8700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "d057124d1d6e4bb1bebb81000776b0cd"
format_stamp: "Formatted at 2026-09-26 04:08:06 on dist-test-slave-xf24"
server_key: "9119bee25e5c621bcc56ec1afdec054c"
server_key_iv: "0f408f79e3e91c2c8fc5f84e5b71464c"
server_key_version: "encryptionkey@0"
I20260926 04:08:06.417402 8700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:06.438642 8700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:06.441346 8700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:06.441534 8700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:06.442162 8700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:06.442205 8700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.442235 8700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:06.442287 8700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.449496 8700 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:33521
I20260926 04:08:06.449596 8830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:33521 every 8 connection(s)
I20260926 04:08:06.449904 8700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:06.454135 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.442997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:51467 (local address 127.0.213.254:41209)
0926 04:08:06.443049 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.443051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.443222 (+ 171us) server_negotiation.cc:407] Connection header received
0926 04:08:06.443948 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.443949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.443967 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.444078 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.445114 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.445981 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.447785 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.448171 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.448259 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.451191 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.451193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.451193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.451201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.452769 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.453184 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.453185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.453185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.453236 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.453505 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.453506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.453506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.453616 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.453671 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.454004 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.454070 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 04:08:06.454423 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.442827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.213.195:51467)
0926 04:08:06.443050 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.443100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.443774 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.444220 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.444227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.444663 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.444976 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.444984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.446098 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.446099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.447681 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.447684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.448268 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.448270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.448452 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.449117 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.449132 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.451092 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.452854 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.452861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.452863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.453114 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.453319 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.453320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.453321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.453438 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.453694 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.453700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.453872 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.453975 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.454122 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
I20260926 04:08:06.455067 8831 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41209
I20260926 04:08:06.455163 8831 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:06.455381 8831 heartbeater.cc:511] Master 127.0.213.254:41209 requested a full tablet report, sending...
I20260926 04:08:06.455818 8347 ts_manager.cc:194] Registered new tserver with Master: d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521)
I20260926 04:08:06.456298 8347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:51467
I20260926 04:08:06.458106 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 8700
I20260926 04:08:06.458204 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:08:06.458396 855 external_mini_cluster.cc:1428] Setting key bb3394c874764831e67cc630d7c62f66
I20260926 04:08:06.459304 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:08:06.462960 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.456361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:37607 (local address 127.0.213.254:41209)
0926 04:08:06.456439 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.456441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.456505 (+ 64us) server_negotiation.cc:407] Connection header received
0926 04:08:06.456581 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.456581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.456590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.456659 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.456982 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.457505 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.458636 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.458881 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.458988 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.460005 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.460006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.460007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.460014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.461666 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.462120 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.462121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.462121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.462181 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.462438 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.462439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.462439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.462563 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.462629 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.462821 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.462896 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:08:06.463080 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.456307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.213.195:37607)
0926 04:08:06.456442 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.456454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.456522 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.456666 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.456667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.456853 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.456907 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.456908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.457618 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.457619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.458496 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.458499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.458990 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.458991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.459062 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.459597 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.459602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.459912 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.461782 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.461783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.461784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.462026 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.462256 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.462257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.462258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.462340 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.462641 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.462642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.462704 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.462789 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.462915 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 04:08:06.482856 8399 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:06.476523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.213.254:41209)
0926 04:08:06.476589 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.476591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.476601 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:06.476632 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.476634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.476651 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.476719 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.477539 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.478084 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.479030 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.479271 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.479359 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.482753 (+ 3394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60690: BlockingRecv error: recv got EOF from 127.0.0.1:60690 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:06.491044 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.483627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.0.213.254:41209)
0926 04:08:06.483693 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.483695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.483764 (+ 69us) server_negotiation.cc:407] Connection header received
0926 04:08:06.483865 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.483866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.483876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.483955 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.484535 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.485131 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.485977 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.486214 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.486288 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.488599 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.488601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.488601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.488609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.489900 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.490234 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.490235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.490235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.490292 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.490511 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.490512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.490512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.490634 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.490740 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.490851 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.490932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:08:06.492950 8347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:08:06.494493 8347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:06.501263 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.497138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.0.213.195:33521)
0926 04:08:06.497399 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.497405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.497514 (+ 109us) server_negotiation.cc:407] Connection header received
0926 04:08:06.497644 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.497647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.497664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.497731 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:06.497983 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.498736 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.499979 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.500823 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.500912 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.500985 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.501125 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.501163 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260926 04:08:06.501253 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.497042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:33521 (local address 127.0.0.1:39044)
0926 04:08:06.497423 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.497451 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.497521 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.497804 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.497807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.497810 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:06.497909 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.497911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.498805 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.498805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.499908 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.499910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.500927 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.500928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.501007 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.501022 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.501100 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.501152 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260926 04:08:06.502075 8765 tablet_service.cc:1467] Processing CreateTablet for tablet deb660a6e44f4dfb85d7975d07be99d6 (DEFAULT_TABLE table=test-table [id=92726d46cbe0484fbc76815021fdd24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:06.502487 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.497042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44941 (local address 127.0.0.1:42832)
0926 04:08:06.497409 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.497451 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.497521 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.497804 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.497807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.497812 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:06.497910 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.497911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.500331 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.500332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.501214 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.501217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.502223 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.502225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.502291 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.502296 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.502348 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.502401 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260926 04:08:06.502527 8765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb660a6e44f4dfb85d7975d07be99d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.502839 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.497375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.213.194:34569)
0926 04:08:06.497549 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.497552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.497779 (+ 227us) server_negotiation.cc:407] Connection header received
0926 04:08:06.497881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.497884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.497902 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.497975 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:06.499072 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.500149 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.501698 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.502467 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.502548 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.502620 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.502704 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.502740 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260926 04:08:06.502812 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.497286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34569 (local address 127.0.0.1:37630)
0926 04:08:06.497622 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.497630 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.497672 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.498853 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.498854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.498856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:06.498932 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.498934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.500232 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.500233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.501593 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.501595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.502561 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.502562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.502628 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.502633 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.502691 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.502732 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":27,"thread_start_us":59,"threads_started":1}
I20260926 04:08:06.503134 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.497238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.0.213.193:44941)
0926 04:08:06.497563 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.497567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.497577 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:06.497620 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.497622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.497639 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.497724 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:06.499024 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.500254 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.501305 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.502028 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.502100 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.502186 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.502846 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.502897 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":85,"threads_started":1}
I20260926 04:08:06.505906 8859 tablet_bootstrap.cc:492] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Bootstrap starting.
I20260926 04:08:06.506697 8859 tablet_bootstrap.cc:654] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.506659 8606 tablet_service.cc:1467] Processing CreateTablet for tablet deb660a6e44f4dfb85d7975d07be99d6 (DEFAULT_TABLE table=test-table [id=92726d46cbe0484fbc76815021fdd24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:06.507045 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb660a6e44f4dfb85d7975d07be99d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.507028 8859 log.cc:824] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:06.507838 8859 tablet_bootstrap.cc:492] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: No bootstrap required, opened a new log
I20260926 04:08:06.507915 8859 ts_tablet_manager.cc:1397] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:08:06.508594 8485 tablet_service.cc:1467] Processing CreateTablet for tablet deb660a6e44f4dfb85d7975d07be99d6 (DEFAULT_TABLE table=test-table [id=92726d46cbe0484fbc76815021fdd24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:06.508956 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb660a6e44f4dfb85d7975d07be99d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.510149 8859 raft_consensus.cc:348] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.510325 8859 raft_consensus.cc:374] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.510371 8859 raft_consensus.cc:729] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.510542 8859 consensus_queue.cc:260] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.510854 8859 ts_tablet_manager.cc:1428] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:08:06.510903 8831 heartbeater.cc:503] Master 127.0.213.254:41209 was elected leader, sending a full tablet report...
I20260926 04:08:06.511437 8862 tablet_bootstrap.cc:492] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Bootstrap starting.
I20260926 04:08:06.512209 8862 tablet_bootstrap.cc:654] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.512681 8862 log.cc:824] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:06.513478 8864 tablet_bootstrap.cc:492] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Bootstrap starting.
I20260926 04:08:06.514070 8862 tablet_bootstrap.cc:492] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: No bootstrap required, opened a new log
I20260926 04:08:06.514155 8862 ts_tablet_manager.cc:1397] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 04:08:06.514449 8864 tablet_bootstrap.cc:654] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.514796 8864 log.cc:824] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:06.515671 8864 tablet_bootstrap.cc:492] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: No bootstrap required, opened a new log
I20260926 04:08:06.515748 8864 ts_tablet_manager.cc:1397] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:08:06.516420 8862 raft_consensus.cc:348] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.516603 8862 raft_consensus.cc:374] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.516640 8862 raft_consensus.cc:729] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.516776 8862 consensus_queue.cc:260] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.517078 8862 ts_tablet_manager.cc:1428] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 04:08:06.517256 8691 heartbeater.cc:503] Master 127.0.213.254:41209 was elected leader, sending a full tablet report...
I20260926 04:08:06.517804 8864 raft_consensus.cc:348] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.518014 8864 raft_consensus.cc:374] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.518074 8864 raft_consensus.cc:729] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f833f6dbcf94eec981d0e950593b4b9, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.518203 8864 consensus_queue.cc:260] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.518496 8864 ts_tablet_manager.cc:1428] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 04:08:06.518519 8551 heartbeater.cc:503] Master 127.0.213.254:41209 was elected leader, sending a full tablet report...
W20260926 04:08:06.531195 8692 tablet.cc:2369] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:08:06.577181 8861 raft_consensus.cc:482] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:06.577319 8861 raft_consensus.cc:504] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.577783 8861 leader_election.cc:288] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:06.581786 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.578192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:59005 (local address 127.0.213.193:44941)
0926 04:08:06.578244 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.578246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.578257 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:08:06.578317 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.578318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.578328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.578381 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:06.578596 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.579338 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.580822 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.581379 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.581462 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.581470 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.581668 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.581704 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 04:08:06.581786 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.578314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34753 (local address 127.0.213.194:34569)
0926 04:08:06.578384 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.578387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.578479 (+ 92us) server_negotiation.cc:407] Connection header received
0926 04:08:06.578530 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.578531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.578538 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.578604 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:06.578816 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.579518 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.580868 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.581379 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.581462 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.581470 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.581667 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.581704 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:08:06.581808 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.578231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34569 (local address 127.0.213.195:34753)
0926 04:08:06.578393 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.578403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.578469 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.578676 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.578677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.578679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:06.578746 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.578747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.579615 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.579616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.580768 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.580770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.581483 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.581484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.581580 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.581584 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.581649 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.581710 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":32,"thread_start_us":116,"threads_started":1}
I20260926 04:08:06.581817 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.578094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44941 (local address 127.0.213.195:59005)
0926 04:08:06.578162 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.578180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.578245 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.578447 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.578449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.578451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:06.578525 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.578527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.579426 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.579427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.580714 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.580717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.581483 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.581484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.581583 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.581584 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.581649 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.581710 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2}
I20260926 04:08:06.582252 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb660a6e44f4dfb85d7975d07be99d6" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" is_pre_election: true
I20260926 04:08:06.582414 8645 raft_consensus.cc:2454] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 0.
I20260926 04:08:06.582509 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb660a6e44f4dfb85d7975d07be99d6" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9" is_pre_election: true
I20260926 04:08:06.582670 8505 raft_consensus.cc:2454] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 0.
I20260926 04:08:06.582639 8718 leader_election.cc:302] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d057124d1d6e4bb1bebb81000776b0cd, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:06.582765 8861 raft_consensus.cc:2781] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:06.582820 8861 raft_consensus.cc:482] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:06.582880 8861 raft_consensus.cc:3037] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:06.583911 8861 raft_consensus.cc:504] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.584084 8861 leader_election.cc:288] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 election: Requested vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:06.584263 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb660a6e44f4dfb85d7975d07be99d6" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf"
I20260926 04:08:06.584311 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb660a6e44f4dfb85d7975d07be99d6" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9"
I20260926 04:08:06.584355 8645 raft_consensus.cc:3037] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:06.584414 8505 raft_consensus.cc:3037] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:06.585070 8645 raft_consensus.cc:2454] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 1.
I20260926 04:08:06.585187 8505 raft_consensus.cc:2454] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 1.
I20260926 04:08:06.585260 8718 leader_election.cc:302] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d057124d1d6e4bb1bebb81000776b0cd, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:06.585373 8861 raft_consensus.cc:2781] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:06.585580 8861 raft_consensus.cc:686] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 1 LEADER]: Becoming Leader. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Running, Role: LEADER
I20260926 04:08:06.585690 8861 consensus_queue.cc:237] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } }
I20260926 04:08:06.586661 8347 catalog_manager.cc:5495] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd reported cstate change: term changed from 0 to 1, leader changed from <none> to d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195). New cstate: current_term: 1 leader_uuid: "d057124d1d6e4bb1bebb81000776b0cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260926 04:08:06.593942 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.591456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.213.195:33521)
0926 04:08:06.591515 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.591517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.591526 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:06.591561 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.591562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.591585 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.591664 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:06.591824 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.592372 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.593147 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.593344 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.593423 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.593602 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:06.593697 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:06.593804 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.593831 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.593852 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 04:08:06.595774 8645 raft_consensus.cc:1261] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Refusing update from remote peer d057124d1d6e4bb1bebb81000776b0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:06.595752 8505 raft_consensus.cc:1261] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Refusing update from remote peer d057124d1d6e4bb1bebb81000776b0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:06.596190 8869 consensus_queue.cc:1035] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:06.596343 8861 consensus_queue.cc:1035] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:06.598754 8872 mvcc.cc:205] Tried to move back new op lower bound from 7333460732293435392 to 7333460732255899648. Current Snapshot: MvccSnapshot[applied={T|T < 7333460732293435392}]
I20260926 04:08:06.599316 8874 mvcc.cc:205] Tried to move back new op lower bound from 7333460732293435392 to 7333460732255899648. Current Snapshot: MvccSnapshot[applied={T|T < 7333460732293435392}]
I20260926 04:08:06.600010 8873 mvcc.cc:205] Tried to move back new op lower bound from 7333460732293435392 to 7333460732255899648. Current Snapshot: MvccSnapshot[applied={T|T < 7333460732293435392 or (T in {7333460732293435392})}]
I20260926 04:08:06.601464 8347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:06.601569 8347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:06.602453 8347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Sending DeleteTablet for 3 replicas of tablet deb660a6e44f4dfb85d7975d07be99d6
I20260926 04:08:06.602813 8765 tablet_service.cc:1514] Processing DeleteTablet for tablet deb660a6e44f4dfb85d7975d07be99d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:06 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:39044
I20260926 04:08:06.602875 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet deb660a6e44f4dfb85d7975d07be99d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:06 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42832
I20260926 04:08:06.603086 8606 tablet_service.cc:1514] Processing DeleteTablet for tablet deb660a6e44f4dfb85d7975d07be99d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:06 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:37630
I20260926 04:08:06.603286 8883 tablet_replica.cc:330] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: stopping tablet replica
I20260926 04:08:06.603133 8882 tablet_replica.cc:330] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: stopping tablet replica
I20260926 04:08:06.603442 8882 raft_consensus.cc:2229] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:06.603396 8347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:08:06.603533 8347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:06.603682 8883 raft_consensus.cc:2229] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:06.603698 8882 raft_consensus.cc:2258] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:06.603889 8883 raft_consensus.cc:2258] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:06.604307 8882 ts_tablet_manager.cc:1905] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:06.604414 8883 ts_tablet_manager.cc:1905] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:06.605419 8485 tablet_service.cc:1467] Processing CreateTablet for tablet 80d968f3bcb244819d4cd610ff4acd7c (DEFAULT_TABLE table=test-table [id=c94c58e2967c43688ea8a02c377c55ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:06.605544 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d968f3bcb244819d4cd610ff4acd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.606300 8765 tablet_service.cc:1467] Processing CreateTablet for tablet 80d968f3bcb244819d4cd610ff4acd7c (DEFAULT_TABLE table=test-table [id=c94c58e2967c43688ea8a02c377c55ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:06.606395 8882 ts_tablet_manager.cc:1918] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:06.606419 8765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d968f3bcb244819d4cd610ff4acd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.606472 8882 log.cc:1196] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/deb660a6e44f4dfb85d7975d07be99d6
I20260926 04:08:06.606732 8882 ts_tablet_manager.cc:1939] T deb660a6e44f4dfb85d7975d07be99d6 P 1f833f6dbcf94eec981d0e950593b4b9: Deleting consensus metadata
I20260926 04:08:06.607049 8864 tablet_bootstrap.cc:492] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Bootstrap starting.
I20260926 04:08:06.607743 8864 tablet_bootstrap.cc:654] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.607811 8859 tablet_bootstrap.cc:492] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Bootstrap starting.
I20260926 04:08:06.608206 8606 tablet_service.cc:1467] Processing CreateTablet for tablet 80d968f3bcb244819d4cd610ff4acd7c (DEFAULT_TABLE table=test-table [id=c94c58e2967c43688ea8a02c377c55ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:06.608261 8859 tablet_bootstrap.cc:654] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.608405 8334 catalog_manager.cc:4841] TS 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941): tablet deb660a6e44f4dfb85d7975d07be99d6 (table test-table [id=92726d46cbe0484fbc76815021fdd24f]) successfully deleted
I20260926 04:08:06.608618 8864 tablet_bootstrap.cc:492] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: No bootstrap required, opened a new log
I20260926 04:08:06.608662 8881 tablet_replica.cc:330] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: stopping tablet replica
I20260926 04:08:06.608664 8864 ts_tablet_manager.cc:1397] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 04:08:06.608765 8881 raft_consensus.cc:2229] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:06.608807 8864 raft_consensus.cc:348] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.608855 8859 tablet_bootstrap.cc:492] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: No bootstrap required, opened a new log
I20260926 04:08:06.608873 8864 raft_consensus.cc:374] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.608896 8864 raft_consensus.cc:729] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f833f6dbcf94eec981d0e950593b4b9, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.608872 8881 raft_consensus.cc:2258] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:06.608896 8859 ts_tablet_manager.cc:1397] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.608943 8864 consensus_queue.cc:260] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.609054 8864 ts_tablet_manager.cc:1428] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.609055 8859 raft_consensus.cc:348] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.609117 8859 raft_consensus.cc:374] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.609138 8859 raft_consensus.cc:729] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.609192 8859 consensus_queue.cc:260] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.609279 8859 ts_tablet_manager.cc:1428] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.609869 8881 ts_tablet_manager.cc:1905] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:06.611414 8881 ts_tablet_manager.cc:1918] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:06.611521 8881 log.cc:1196] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/deb660a6e44f4dfb85d7975d07be99d6
I20260926 04:08:06.611827 8881 ts_tablet_manager.cc:1939] T deb660a6e44f4dfb85d7975d07be99d6 P d057124d1d6e4bb1bebb81000776b0cd: Deleting consensus metadata
I20260926 04:08:06.612355 8332 catalog_manager.cc:4841] TS d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521): tablet deb660a6e44f4dfb85d7975d07be99d6 (table test-table [id=92726d46cbe0484fbc76815021fdd24f]) successfully deleted
I20260926 04:08:06.614437 8883 ts_tablet_manager.cc:1918] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:06.614440 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d968f3bcb244819d4cd610ff4acd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.614569 8883 log.cc:1196] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/deb660a6e44f4dfb85d7975d07be99d6
I20260926 04:08:06.614846 8883 ts_tablet_manager.cc:1939] T deb660a6e44f4dfb85d7975d07be99d6 P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting consensus metadata
I20260926 04:08:06.615753 8862 tablet_bootstrap.cc:492] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Bootstrap starting.
I20260926 04:08:06.615793 8334 catalog_manager.cc:4841] TS fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569): tablet deb660a6e44f4dfb85d7975d07be99d6 (table test-table [id=92726d46cbe0484fbc76815021fdd24f]) successfully deleted
I20260926 04:08:06.616256 8862 tablet_bootstrap.cc:654] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.616921 8862 tablet_bootstrap.cc:492] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: No bootstrap required, opened a new log
I20260926 04:08:06.616976 8862 ts_tablet_manager.cc:1397] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.617132 8862 raft_consensus.cc:348] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.617218 8862 raft_consensus.cc:374] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.617238 8862 raft_consensus.cc:729] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.617302 8862 consensus_queue.cc:260] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.617396 8862 ts_tablet_manager.cc:1428] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 04:08:06.700582 8832 tablet.cc:2369] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:08:06.773645 8861 raft_consensus.cc:482] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:06.773768 8861 raft_consensus.cc:504] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.773944 8861 leader_election.cc:288] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:06.774303 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d968f3bcb244819d4cd610ff4acd7c" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9" is_pre_election: true
I20260926 04:08:06.774391 8505 raft_consensus.cc:2454] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 0.
I20260926 04:08:06.774324 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d968f3bcb244819d4cd610ff4acd7c" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" is_pre_election: true
I20260926 04:08:06.774464 8645 raft_consensus.cc:2454] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 0.
I20260926 04:08:06.774559 8718 leader_election.cc:302] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f833f6dbcf94eec981d0e950593b4b9, d057124d1d6e4bb1bebb81000776b0cd; no voters:
I20260926 04:08:06.774698 8861 raft_consensus.cc:2781] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:06.774741 8861 raft_consensus.cc:482] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:06.774811 8861 raft_consensus.cc:3037] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:06.775534 8861 raft_consensus.cc:504] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.775672 8861 leader_election.cc:288] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 election: Requested vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:06.775832 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d968f3bcb244819d4cd610ff4acd7c" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9"
I20260926 04:08:06.775940 8505 raft_consensus.cc:3037] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:06.775933 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d968f3bcb244819d4cd610ff4acd7c" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf"
I20260926 04:08:06.775997 8645 raft_consensus.cc:3037] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:06.776760 8505 raft_consensus.cc:2454] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 1.
I20260926 04:08:06.776755 8645 raft_consensus.cc:2454] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 1.
I20260926 04:08:06.776959 8718 leader_election.cc:302] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f833f6dbcf94eec981d0e950593b4b9, d057124d1d6e4bb1bebb81000776b0cd; no voters:
I20260926 04:08:06.777073 8861 raft_consensus.cc:2781] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:06.777140 8861 raft_consensus.cc:686] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 1 LEADER]: Becoming Leader. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Running, Role: LEADER
I20260926 04:08:06.777191 8861 consensus_queue.cc:237] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.777812 8347 catalog_manager.cc:5495] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd reported cstate change: term changed from 0 to 1, leader changed from <none> to d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195). New cstate: current_term: 1 leader_uuid: "d057124d1d6e4bb1bebb81000776b0cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } health_report { overall_health: HEALTHY } } }
Sep 26 04:08:06 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395685, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.254@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:06.832525 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.824949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41209 (local address 127.0.0.1:60700)
0926 04:08:06.825129 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:06.825148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:06.825247 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:06.825400 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:06.825401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:06.825712 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:06.825759 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.825761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.826436 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.826437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.826976 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:06.826979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.827392 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.827393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:06.827445 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.828035 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:06.828040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:06.829540 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:06.831211 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.831221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.831223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.831463 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.831674 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:06.831675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:06.831675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:06.831737 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:06.832106 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:06.832114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:06.832180 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 04:08:06.832276 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 04:08:06.832405 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260926 04:08:06.832624 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:06.825022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.213.254:41209)
0926 04:08:06.825108 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:06.825112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:06.825220 (+ 108us) server_negotiation.cc:407] Connection header received
0926 04:08:06.825325 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:06.825326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:06.825336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:06.825395 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:06.825853 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.826351 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.827114 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:06.827306 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:06.827391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:06.829644 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:06.829645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:06.829646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:06.829653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:06.831105 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.831544 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.831546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.831546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.831606 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:06.831802 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:06.831803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:06.831803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:06.831956 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:06.832043 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:06.832309 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 04:08:06.832490 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":55}
I20260926 04:08:06.835098 8347 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:60700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 04:08:06.835366 8347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:06.837491 8606 tablet_service.cc:1467] Processing CreateTablet for tablet d3fc6b47055e451290f6d64cf71e775e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=07fef52b5d6e48f3926a6d160eb98a96]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:08:06.837499 8485 tablet_service.cc:1467] Processing CreateTablet for tablet d3fc6b47055e451290f6d64cf71e775e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=07fef52b5d6e48f3926a6d160eb98a96]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:08:06.837671 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fc6b47055e451290f6d64cf71e775e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.837692 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fc6b47055e451290f6d64cf71e775e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.838999 8864 tablet_bootstrap.cc:492] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: Bootstrap starting.
I20260926 04:08:06.839123 8765 tablet_service.cc:1467] Processing CreateTablet for tablet d3fc6b47055e451290f6d64cf71e775e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=07fef52b5d6e48f3926a6d160eb98a96]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 04:08:06.839304 8765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3fc6b47055e451290f6d64cf71e775e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:06.839444 8864 tablet_bootstrap.cc:654] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.840181 8862 tablet_bootstrap.cc:492] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: Bootstrap starting.
I20260926 04:08:06.840201 8864 tablet_bootstrap.cc:492] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: No bootstrap required, opened a new log
I20260926 04:08:06.840267 8864 ts_tablet_manager.cc:1397] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:06.840408 8864 raft_consensus.cc:348] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.840474 8864 raft_consensus.cc:374] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.840497 8864 raft_consensus.cc:729] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f833f6dbcf94eec981d0e950593b4b9, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.840551 8864 consensus_queue.cc:260] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.840632 8862 tablet_bootstrap.cc:654] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.840649 8864 ts_tablet_manager.cc:1428] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.840723 8867 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:06.840843 8867 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:08:06.841005 8859 tablet_bootstrap.cc:492] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: Bootstrap starting.
I20260926 04:08:06.841346 8859 tablet_bootstrap.cc:654] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:06.841888 8859 tablet_bootstrap.cc:492] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: No bootstrap required, opened a new log
I20260926 04:08:06.841944 8859 ts_tablet_manager.cc:1397] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:06.842036 8862 tablet_bootstrap.cc:492] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: No bootstrap required, opened a new log
I20260926 04:08:06.842096 8862 ts_tablet_manager.cc:1397] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:08:06.842060 8859 raft_consensus.cc:348] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.842146 8859 raft_consensus.cc:374] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.842176 8859 raft_consensus.cc:729] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.842226 8859 consensus_queue.cc:260] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.842252 8862 raft_consensus.cc:348] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.842316 8859 ts_tablet_manager.cc:1428] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.842327 8862 raft_consensus.cc:374] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:06.842330 8861 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:06.842350 8862 raft_consensus.cc:729] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Initialized, Role: FOLLOWER
I20260926 04:08:06.842394 8861 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:08:06.842427 8862 consensus_queue.cc:260] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:06.842527 8862 ts_tablet_manager.cc:1428] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:06.842557 8866 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:06.842667 8866 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 04:08:06.850646 8552 tablet.cc:2369] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:08:07.223831 8866 raft_consensus.cc:482] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:07.223942 8866 raft_consensus.cc:504] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.224406 8866 leader_election.cc:288] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521)
I20260926 04:08:07.227511 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.224694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:57147 (local address 127.0.213.193:44941)
0926 04:08:07.224805 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.224806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.224816 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:07.224891 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.224892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.224904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.224986 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:07.225138 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.225641 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.226738 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.227132 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.227232 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.227242 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.227385 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.227432 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260926 04:08:07.227525 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.224609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44941 (local address 127.0.213.194:57147)
0926 04:08:07.224755 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.224766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:07.224827 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:07.224988 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:07.224989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:07.224991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:07.225059 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.225061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.225741 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.225743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.226638 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.226640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.227231 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.227232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.227306 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.227310 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:07.227380 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 04:08:07.227424 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":66,"thread_start_us":63,"threads_started":1}
I20260926 04:08:07.227612 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.224596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:33521 (local address 127.0.213.194:49143)
0926 04:08:07.224748 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.224761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:07.224863 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:07.225052 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:07.225053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:07.225056 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:07.225123 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.225125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.225817 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.225818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.226706 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.226708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.227292 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.227294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.227390 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.227394 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:07.227459 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:08:07.227501 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":66,"thread_start_us":68,"threads_started":1}
I20260926 04:08:07.227603 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.224657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:49143 (local address 127.0.213.195:33521)
0926 04:08:07.224752 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.224754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.224846 (+ 92us) server_negotiation.cc:407] Connection header received
0926 04:08:07.224944 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.224946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.224957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.225051 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:07.225212 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.225712 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.226805 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.227203 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.227293 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.227302 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.227480 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.227526 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":41,"threads_started":1}
I20260926 04:08:07.227905 8785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fc6b47055e451290f6d64cf71e775e" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d057124d1d6e4bb1bebb81000776b0cd" is_pre_election: true
I20260926 04:08:07.227967 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fc6b47055e451290f6d64cf71e775e" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9" is_pre_election: true
I20260926 04:08:07.228077 8785 raft_consensus.cc:2454] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 0.
I20260926 04:08:07.228099 8505 raft_consensus.cc:2454] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 0.
I20260926 04:08:07.228251 8576 leader_election.cc:302] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d057124d1d6e4bb1bebb81000776b0cd, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:07.228382 8866 raft_consensus.cc:2781] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:07.228420 8866 raft_consensus.cc:482] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:07.228446 8866 raft_consensus.cc:3037] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.229143 8866 raft_consensus.cc:504] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.229305 8866 leader_election.cc:288] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 election: Requested vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521)
I20260926 04:08:07.229483 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fc6b47055e451290f6d64cf71e775e" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9"
I20260926 04:08:07.229573 8505 raft_consensus.cc:3037] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.230224 8505 raft_consensus.cc:2454] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 1.
I20260926 04:08:07.230374 8785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fc6b47055e451290f6d64cf71e775e" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d057124d1d6e4bb1bebb81000776b0cd"
I20260926 04:08:07.230479 8785 raft_consensus.cc:3037] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.230494 8578 leader_election.cc:302] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f833f6dbcf94eec981d0e950593b4b9, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:07.230614 8866 raft_consensus.cc:2781] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:07.230834 8866 raft_consensus.cc:686] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 LEADER]: Becoming Leader. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Running, Role: LEADER
I20260926 04:08:07.230939 8866 consensus_queue.cc:237] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.231469 8785 raft_consensus.cc:2454] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 1.
I20260926 04:08:07.231484 8901 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: TxnStatusTablet state changed. Reason: New leader fb912d2079654ebe8f0e77ca68f7c4cf. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.231580 8901 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:08:07.231703 8347 catalog_manager.cc:5495] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf reported cstate change: term changed from 0 to 1, leader changed from <none> to fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194). New cstate: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } health_report { overall_health: UNKNOWN } } }
I20260926 04:08:07.231886 8903 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 04:08:07.232016 8903 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 04:08:07.284569 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.280994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.0.213.194:34569)
0926 04:08:07.281163 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.281166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.281216 (+ 50us) server_negotiation.cc:407] Connection header received
0926 04:08:07.281312 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.281313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.281330 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.281413 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:07.281556 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.282372 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.283687 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.283931 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.284000 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.284230 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.284328 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.284390 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.284472 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.284492 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260926 04:08:07.284613 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.280923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34569 (local address 127.0.0.1:37634)
0926 04:08:07.281162 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.281174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:07.281239 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:07.281427 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:07.281428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:07.281429 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:08:07.281485 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.281487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.282465 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.282466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.283559 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.283563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.284024 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.284025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.284117 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.284137 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.284402 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.284411 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 04:08:07.284468 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:08:07.284524 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260926 04:08:07.286037 8505 raft_consensus.cc:1261] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Refusing update from remote peer fb912d2079654ebe8f0e77ca68f7c4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.286083 8785 raft_consensus.cc:1261] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Refusing update from remote peer fb912d2079654ebe8f0e77ca68f7c4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.286315 8901 consensus_queue.cc:1035] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.286475 8866 consensus_queue.cc:1035] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.287204 8867 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: TxnStatusTablet state changed. Reason: New leader fb912d2079654ebe8f0e77ca68f7c4cf. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.287312 8867 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:08:07.287390 8861 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: TxnStatusTablet state changed. Reason: New leader fb912d2079654ebe8f0e77ca68f7c4cf. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.287478 8861 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:08:07.287925 8866 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.288017 8866 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:08:07.288156 8861 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.288234 8861 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P d057124d1d6e4bb1bebb81000776b0cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:08:07.288965 8866 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.289057 8866 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P fb912d2079654ebe8f0e77ca68f7c4cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 04:08:07.289374 8867 tablet_replica.cc:439] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } } }
I20260926 04:08:07.289467 8867 tablet_replica.cc:442] T d3fc6b47055e451290f6d64cf71e775e P 1f833f6dbcf94eec981d0e950593b4b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 04:08:07.295768 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.292944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34217 (local address 127.0.213.194:34569)
0926 04:08:07.292991 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.292993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.293138 (+ 145us) server_negotiation.cc:407] Connection header received
0926 04:08:07.293244 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.293245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.293257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.293331 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:07.293427 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.294167 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.294961 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.295264 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.295363 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.295508 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.295554 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.295610 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.295657 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.295686 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:08:07.295761 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.292861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34569 (local address 127.0.213.195:34217)
0926 04:08:07.293064 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.293073 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:07.293126 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:07.293335 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:07.293336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:07.293338 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:08:07.293385 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.293386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.294260 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.294261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.294869 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.294871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.295364 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.295365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.295412 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.295426 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.295617 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.295618 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 04:08:07.295653 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 04:08:07.295696 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260926 04:08:07.298602 8505 raft_consensus.cc:1261] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Refusing update from remote peer d057124d1d6e4bb1bebb81000776b0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.298844 8919 consensus_queue.cc:1035] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.298956 8645 raft_consensus.cc:1261] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Refusing update from remote peer d057124d1d6e4bb1bebb81000776b0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.299161 8919 consensus_queue.cc:1035] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.308018 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.305289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:59505 (local address 127.0.213.195:33521)
0926 04:08:07.305345 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.305347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.305416 (+ 69us) server_negotiation.cc:407] Connection header received
0926 04:08:07.305484 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.305484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.305495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.305551 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:07.305750 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.306236 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.307256 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.307467 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.307551 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.307752 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.307795 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.307857 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.307913 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.307936 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:08:07.308045 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.305168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:33521 (local address 127.0.213.194:59505)
0926 04:08:07.305346 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.305358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:07.305428 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:07.305554 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:07.305554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:07.305556 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:08:07.305628 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.305630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.306322 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.306323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.307164 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.307167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.307571 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.307572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.307647 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.307661 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.307859 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.307861 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:08:07.307909 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 04:08:07.307954 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260926 04:08:07.316973 8347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:07.317126 8347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:07.317853 8347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Sending DeleteTablet for 3 replicas of tablet 80d968f3bcb244819d4cd610ff4acd7c
I20260926 04:08:07.318179 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d968f3bcb244819d4cd610ff4acd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42832
I20260926 04:08:07.318221 8606 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d968f3bcb244819d4cd610ff4acd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:37630
I20260926 04:08:07.318431 8765 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d968f3bcb244819d4cd610ff4acd7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:39044
I20260926 04:08:07.318517 8926 tablet_replica.cc:330] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: stopping tablet replica
I20260926 04:08:07.318603 8926 raft_consensus.cc:2229] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:07.318660 8926 raft_consensus.cc:2258] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.318688 8927 tablet_replica.cc:330] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: stopping tablet replica
I20260926 04:08:07.318765 8925 tablet_replica.cc:330] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: stopping tablet replica
I20260926 04:08:07.318792 8927 raft_consensus.cc:2229] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:07.318815 8925 raft_consensus.cc:2229] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:07.318851 8925 raft_consensus.cc:2258] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.318863 8927 raft_consensus.cc:2258] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.319101 8927 ts_tablet_manager.cc:1905] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.322168 8926 ts_tablet_manager.cc:1905] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.322170 8925 ts_tablet_manager.cc:1905] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.324353 8925 ts_tablet_manager.cc:1918] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:08:07.324479 8925 log.cc:1196] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/80d968f3bcb244819d4cd610ff4acd7c
I20260926 04:08:07.324777 8925 ts_tablet_manager.cc:1939] T 80d968f3bcb244819d4cd610ff4acd7c P 1f833f6dbcf94eec981d0e950593b4b9: Deleting consensus metadata
I20260926 04:08:07.325016 8926 ts_tablet_manager.cc:1918] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:08:07.325203 8334 catalog_manager.cc:4841] TS 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941): tablet 80d968f3bcb244819d4cd610ff4acd7c (table test-table [id=c94c58e2967c43688ea8a02c377c55ef]) successfully deleted
I20260926 04:08:07.325248 8926 log.cc:1196] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/80d968f3bcb244819d4cd610ff4acd7c
I20260926 04:08:07.325524 8926 ts_tablet_manager.cc:1939] T 80d968f3bcb244819d4cd610ff4acd7c P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting consensus metadata
I20260926 04:08:07.325874 8334 catalog_manager.cc:4841] TS fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569): tablet 80d968f3bcb244819d4cd610ff4acd7c (table test-table [id=c94c58e2967c43688ea8a02c377c55ef]) successfully deleted
I20260926 04:08:07.328230 8927 ts_tablet_manager.cc:1918] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:08:07.328318 8927 log.cc:1196] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/80d968f3bcb244819d4cd610ff4acd7c
I20260926 04:08:07.328584 8927 ts_tablet_manager.cc:1939] T 80d968f3bcb244819d4cd610ff4acd7c P d057124d1d6e4bb1bebb81000776b0cd: Deleting consensus metadata
I20260926 04:08:07.328971 8332 catalog_manager.cc:4841] TS d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521): tablet 80d968f3bcb244819d4cd610ff4acd7c (table test-table [id=c94c58e2967c43688ea8a02c377c55ef]) successfully deleted
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 04:08:07.346140 8935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:07.339024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.213.254:41209)
0926 04:08:07.339194 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.339197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.339394 (+ 197us) server_negotiation.cc:407] Connection header received
0926 04:08:07.339426 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.339427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.339439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.339502 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:07.340376 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.341027 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.342185 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.342409 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.342516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.346010 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60716: BlockingRecv error: recv got EOF from 127.0.0.1:60716 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM
I20260926 04:08:07.355795 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.347130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.0.213.254:41209)
0926 04:08:07.347187 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.347190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.347218 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:08:07.347293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.347294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.347306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.347362 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:07.347910 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.348773 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.349508 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.349787 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.349867 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.352682 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:07.352684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:07.352684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:07.352692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:07.354425 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:07.354835 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:07.354835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:07.354836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:07.354894 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:07.355194 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:07.355196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:07.355196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:07.355333 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:07.355431 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.355611 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.355689 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:08:07.357803 8347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:08:07.357996 8347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:07.359477 8765 tablet_service.cc:1467] Processing CreateTablet for tablet f7459e979a994fb497586cade1fd5c7e (DEFAULT_TABLE table=test-table [id=2068fb743f154363a653d66c4d7f85be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:07.359637 8765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7459e979a994fb497586cade1fd5c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:07.359740 8606 tablet_service.cc:1467] Processing CreateTablet for tablet f7459e979a994fb497586cade1fd5c7e (DEFAULT_TABLE table=test-table [id=2068fb743f154363a653d66c4d7f85be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:07.359886 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7459e979a994fb497586cade1fd5c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:07.361042 8941 tablet_bootstrap.cc:492] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Bootstrap starting.
I20260926 04:08:07.361323 8485 tablet_service.cc:1467] Processing CreateTablet for tablet f7459e979a994fb497586cade1fd5c7e (DEFAULT_TABLE table=test-table [id=2068fb743f154363a653d66c4d7f85be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:07.361387 8942 tablet_bootstrap.cc:492] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Bootstrap starting.
I20260926 04:08:07.361447 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7459e979a994fb497586cade1fd5c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:07.361866 8942 tablet_bootstrap.cc:654] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:07.361861 8941 tablet_bootstrap.cc:654] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:07.362486 8941 tablet_bootstrap.cc:492] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: No bootstrap required, opened a new log
I20260926 04:08:07.362494 8942 tablet_bootstrap.cc:492] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: No bootstrap required, opened a new log
I20260926 04:08:07.362536 8941 ts_tablet_manager.cc:1397] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:08:07.362536 8942 ts_tablet_manager.cc:1397] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:07.362841 8941 raft_consensus.cc:348] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.362850 8942 raft_consensus.cc:348] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.362929 8941 raft_consensus.cc:374] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:07.362954 8942 raft_consensus.cc:374] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:07.362958 8941 raft_consensus.cc:729] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Initialized, Role: FOLLOWER
I20260926 04:08:07.362979 8942 raft_consensus.cc:729] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Initialized, Role: FOLLOWER
I20260926 04:08:07.363009 8941 consensus_queue.cc:260] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.363024 8942 consensus_queue.cc:260] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.363109 8941 ts_tablet_manager.cc:1428] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:07.363147 8942 ts_tablet_manager.cc:1428] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:07.363766 8945 tablet_bootstrap.cc:492] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Bootstrap starting.
I20260926 04:08:07.364265 8945 tablet_bootstrap.cc:654] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:07.364919 8945 tablet_bootstrap.cc:492] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: No bootstrap required, opened a new log
I20260926 04:08:07.364987 8945 ts_tablet_manager.cc:1397] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:07.365150 8945 raft_consensus.cc:348] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.365234 8945 raft_consensus.cc:374] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:07.365262 8945 raft_consensus.cc:729] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f833f6dbcf94eec981d0e950593b4b9, State: Initialized, Role: FOLLOWER
I20260926 04:08:07.365311 8945 consensus_queue.cc:260] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.365409 8945 ts_tablet_manager.cc:1428] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:07.392653 8861 raft_consensus.cc:482] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:07.392776 8861 raft_consensus.cc:504] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.392931 8861 leader_election.cc:288] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:07.393260 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7459e979a994fb497586cade1fd5c7e" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" is_pre_election: true
I20260926 04:08:07.393282 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7459e979a994fb497586cade1fd5c7e" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9" is_pre_election: true
I20260926 04:08:07.393379 8505 raft_consensus.cc:2454] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 0.
I20260926 04:08:07.393380 8645 raft_consensus.cc:2454] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 0.
I20260926 04:08:07.393610 8718 leader_election.cc:302] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d057124d1d6e4bb1bebb81000776b0cd, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:07.393783 8861 raft_consensus.cc:2781] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:07.393862 8861 raft_consensus.cc:482] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:07.393906 8861 raft_consensus.cc:3037] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.394584 8861 raft_consensus.cc:504] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.394754 8861 leader_election.cc:288] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 election: Requested vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569)
I20260926 04:08:07.395007 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7459e979a994fb497586cade1fd5c7e" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf"
I20260926 04:08:07.395007 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7459e979a994fb497586cade1fd5c7e" candidate_uuid: "d057124d1d6e4bb1bebb81000776b0cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9"
I20260926 04:08:07.395110 8645 raft_consensus.cc:3037] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.395135 8505 raft_consensus.cc:3037] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.395905 8505 raft_consensus.cc:2454] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 1.
I20260926 04:08:07.395957 8645 raft_consensus.cc:2454] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d057124d1d6e4bb1bebb81000776b0cd in term 1.
I20260926 04:08:07.396149 8718 leader_election.cc:302] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f833f6dbcf94eec981d0e950593b4b9, d057124d1d6e4bb1bebb81000776b0cd; no voters:
I20260926 04:08:07.396263 8861 raft_consensus.cc:2781] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:07.396322 8861 raft_consensus.cc:686] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 1 LEADER]: Becoming Leader. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Running, Role: LEADER
I20260926 04:08:07.396384 8861 consensus_queue.cc:237] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.397208 8346 catalog_manager.cc:5495] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd reported cstate change: term changed from 0 to 1, leader changed from <none> to d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195). New cstate: current_term: 1 leader_uuid: "d057124d1d6e4bb1bebb81000776b0cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } health_report { overall_health: HEALTHY } } }
I20260926 04:08:07.412012 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.409683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.0.213.195:33521)
0926 04:08:07.409757 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.409759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.409766 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:07.409787 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.409787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.409799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.409866 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:07.410009 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.410517 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.411286 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.411482 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.411562 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.411729 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.411765 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.411827 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.411891 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.411916 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:08:07.412892 8645 raft_consensus.cc:1261] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Refusing update from remote peer d057124d1d6e4bb1bebb81000776b0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.412894 8505 raft_consensus.cc:1261] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Refusing update from remote peer d057124d1d6e4bb1bebb81000776b0cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.413160 8919 consensus_queue.cc:1035] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.413384 8861 consensus_queue.cc:1035] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.416000 8346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:07.416136 8346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:07.416775 8346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Sending DeleteTablet for 3 replicas of tablet f7459e979a994fb497586cade1fd5c7e
I20260926 04:08:07.417016 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet f7459e979a994fb497586cade1fd5c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42832
I20260926 04:08:07.417104 8606 tablet_service.cc:1514] Processing DeleteTablet for tablet f7459e979a994fb497586cade1fd5c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:37630
I20260926 04:08:07.417115 8765 tablet_service.cc:1514] Processing DeleteTablet for tablet f7459e979a994fb497586cade1fd5c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:39044
I20260926 04:08:07.417209 8926 tablet_replica.cc:330] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: stopping tablet replica
I20260926 04:08:07.417209 8925 tablet_replica.cc:330] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: stopping tablet replica
I20260926 04:08:07.417236 8927 tablet_replica.cc:330] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: stopping tablet replica
I20260926 04:08:07.417269 8925 raft_consensus.cc:2229] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:07.417266 8926 raft_consensus.cc:2229] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:07.417286 8927 raft_consensus.cc:2229] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:07.417315 8925 raft_consensus.cc:2258] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.417320 8926 raft_consensus.cc:2258] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.417339 8927 raft_consensus.cc:2258] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.417510 8346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 04:08:07.417631 8926 ts_tablet_manager.cc:1905] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 04:08:07.417649 8346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:07.417764 8927 ts_tablet_manager.cc:1905] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.418056 8925 ts_tablet_manager.cc:1905] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.419191 8485 tablet_service.cc:1467] Processing CreateTablet for tablet d769d9e140ad47baa3008973f5d5f337 (DEFAULT_TABLE table=test-table [id=8039c0153b1a4a4da8ca6e9086cc8684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:07.419313 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d769d9e140ad47baa3008973f5d5f337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:07.419543 8606 tablet_service.cc:1467] Processing CreateTablet for tablet d769d9e140ad47baa3008973f5d5f337 (DEFAULT_TABLE table=test-table [id=8039c0153b1a4a4da8ca6e9086cc8684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:07.419668 8925 ts_tablet_manager.cc:1918] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:07.419727 8925 log.cc:1196] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/f7459e979a994fb497586cade1fd5c7e
I20260926 04:08:07.419921 8927 ts_tablet_manager.cc:1918] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:07.419968 8925 ts_tablet_manager.cc:1939] T f7459e979a994fb497586cade1fd5c7e P 1f833f6dbcf94eec981d0e950593b4b9: Deleting consensus metadata
I20260926 04:08:07.419930 8765 tablet_service.cc:1467] Processing CreateTablet for tablet d769d9e140ad47baa3008973f5d5f337 (DEFAULT_TABLE table=test-table [id=8039c0153b1a4a4da8ca6e9086cc8684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:07.419986 8927 log.cc:1196] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/f7459e979a994fb497586cade1fd5c7e
I20260926 04:08:07.420089 8765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d769d9e140ad47baa3008973f5d5f337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:07.420190 8926 ts_tablet_manager.cc:1918] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:07.420241 8926 log.cc:1196] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/f7459e979a994fb497586cade1fd5c7e
I20260926 04:08:07.420341 8334 catalog_manager.cc:4841] TS 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941): tablet f7459e979a994fb497586cade1fd5c7e (table test-table [id=2068fb743f154363a653d66c4d7f85be]) successfully deleted
I20260926 04:08:07.420454 8926 ts_tablet_manager.cc:1939] T f7459e979a994fb497586cade1fd5c7e P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting consensus metadata
I20260926 04:08:07.420622 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d769d9e140ad47baa3008973f5d5f337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:07.420622 8945 tablet_bootstrap.cc:492] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9: Bootstrap starting.
I20260926 04:08:07.421025 8945 tablet_bootstrap.cc:654] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:07.421234 8941 tablet_bootstrap.cc:492] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Bootstrap starting.
I20260926 04:08:07.421569 8941 tablet_bootstrap.cc:654] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:07.421635 8945 tablet_bootstrap.cc:492] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9: No bootstrap required, opened a new log
I20260926 04:08:07.421701 8945 ts_tablet_manager.cc:1397] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:07.421868 8945 raft_consensus.cc:348] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.421948 8942 tablet_bootstrap.cc:492] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Bootstrap starting.
I20260926 04:08:07.421953 8945 raft_consensus.cc:374] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:07.421979 8945 raft_consensus.cc:729] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f833f6dbcf94eec981d0e950593b4b9, State: Initialized, Role: FOLLOWER
I20260926 04:08:07.422024 8945 consensus_queue.cc:260] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.422125 8945 ts_tablet_manager.cc:1428] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:07.422207 8941 tablet_bootstrap.cc:492] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: No bootstrap required, opened a new log
I20260926 04:08:07.422256 8942 tablet_bootstrap.cc:654] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:07.422261 8941 ts_tablet_manager.cc:1397] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:07.422392 8941 raft_consensus.cc:348] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.422464 8941 raft_consensus.cc:374] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:07.422488 8941 raft_consensus.cc:729] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d057124d1d6e4bb1bebb81000776b0cd, State: Initialized, Role: FOLLOWER
I20260926 04:08:07.422519 8941 consensus_queue.cc:260] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.422621 8941 ts_tablet_manager.cc:1428] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:07.422803 8927 ts_tablet_manager.cc:1939] T f7459e979a994fb497586cade1fd5c7e P d057124d1d6e4bb1bebb81000776b0cd: Deleting consensus metadata
I20260926 04:08:07.423090 8332 catalog_manager.cc:4841] TS d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521): tablet f7459e979a994fb497586cade1fd5c7e (table test-table [id=2068fb743f154363a653d66c4d7f85be]) successfully deleted
I20260926 04:08:07.423285 8334 catalog_manager.cc:4841] TS fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569): tablet f7459e979a994fb497586cade1fd5c7e (table test-table [id=2068fb743f154363a653d66c4d7f85be]) successfully deleted
I20260926 04:08:07.423499 8942 tablet_bootstrap.cc:492] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: No bootstrap required, opened a new log
I20260926 04:08:07.423565 8942 ts_tablet_manager.cc:1397] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:08:07.423684 8942 raft_consensus.cc:348] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.423765 8942 raft_consensus.cc:374] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:07.423789 8942 raft_consensus.cc:729] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Initialized, Role: FOLLOWER
I20260926 04:08:07.423842 8942 consensus_queue.cc:260] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.423935 8942 ts_tablet_manager.cc:1428] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:07.610523 8901 raft_consensus.cc:482] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:07.610680 8901 raft_consensus.cc:504] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.610908 8901 leader_election.cc:288] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521)
I20260926 04:08:07.611121 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d769d9e140ad47baa3008973f5d5f337" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9" is_pre_election: true
I20260926 04:08:07.611218 8505 raft_consensus.cc:2454] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 0.
I20260926 04:08:07.611295 8785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d769d9e140ad47baa3008973f5d5f337" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d057124d1d6e4bb1bebb81000776b0cd" is_pre_election: true
I20260926 04:08:07.611399 8785 raft_consensus.cc:2454] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 0.
I20260926 04:08:07.611433 8578 leader_election.cc:302] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f833f6dbcf94eec981d0e950593b4b9, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:07.611570 8901 raft_consensus.cc:2781] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:07.611610 8901 raft_consensus.cc:482] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:07.611632 8901 raft_consensus.cc:3037] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.612358 8901 raft_consensus.cc:504] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.612519 8901 leader_election.cc:288] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 election: Requested vote from peers 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941), d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521)
I20260926 04:08:07.612685 8505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d769d9e140ad47baa3008973f5d5f337" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f833f6dbcf94eec981d0e950593b4b9"
I20260926 04:08:07.612787 8505 raft_consensus.cc:3037] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.612771 8785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d769d9e140ad47baa3008973f5d5f337" candidate_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d057124d1d6e4bb1bebb81000776b0cd"
I20260926 04:08:07.612866 8785 raft_consensus.cc:3037] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:07.613373 8505 raft_consensus.cc:2454] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 1.
I20260926 04:08:07.613574 8785 raft_consensus.cc:2454] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb912d2079654ebe8f0e77ca68f7c4cf in term 1.
I20260926 04:08:07.613577 8578 leader_election.cc:302] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f833f6dbcf94eec981d0e950593b4b9, fb912d2079654ebe8f0e77ca68f7c4cf; no voters:
I20260926 04:08:07.613677 8901 raft_consensus.cc:2781] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:07.613745 8901 raft_consensus.cc:686] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 LEADER]: Becoming Leader. State: Replica: fb912d2079654ebe8f0e77ca68f7c4cf, State: Running, Role: LEADER
I20260926 04:08:07.613803 8901 consensus_queue.cc:237] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } }
I20260926 04:08:07.614401 8347 catalog_manager.cc:5495] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf reported cstate change: term changed from 0 to 1, leader changed from <none> to fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194). New cstate: current_term: 1 leader_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb912d2079654ebe8f0e77ca68f7c4cf" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 } health_report { overall_health: UNKNOWN } } }
I20260926 04:08:07.658191 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.655859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.213.194:34569)
0926 04:08:07.655921 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.655923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.655935 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:08:07.656023 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.656023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.656035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.656137 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:07.656235 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.656725 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.657520 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.657738 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.657796 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.657938 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.657971 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.658007 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.658113 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.658131 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:08:07.662707 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.660639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:44673 (local address 127.0.213.194:34569)
0926 04:08:07.660692 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:07.660694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:07.660701 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:07.660749 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:07.660749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:07.660757 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:07.660802 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:07.660947 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.661380 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.662083 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.662255 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.662313 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.662455 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.662477 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.662514 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:07.662615 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 04:08:07.662630 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260926 04:08:07.662719 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.660522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:34569 (local address 127.0.213.194:44673)
0926 04:08:07.660633 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.660646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:07.660706 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:07.660850 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:07.660850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:07.660852 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 04:08:07.660884 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.660885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.661448 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.661449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.662004 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:07.662005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:07.662342 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:07.662343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:07.662387 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:07.662392 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 04:08:07.662532 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 04:08:07.662534 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:08:07.662613 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 04:08:07.662652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":1}
I20260926 04:08:07.665053 8505 raft_consensus.cc:1261] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Refusing update from remote peer fb912d2079654ebe8f0e77ca68f7c4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.665083 8785 raft_consensus.cc:1261] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Refusing update from remote peer fb912d2079654ebe8f0e77ca68f7c4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:07.665304 8914 consensus_queue.cc:1035] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d057124d1d6e4bb1bebb81000776b0cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.665406 8866 consensus_queue.cc:1035] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f833f6dbcf94eec981d0e950593b4b9" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:07.675701 8347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:07.675830 8347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:07.676502 8347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b12717eb25b74b7c93defae73697c20e: Sending DeleteTablet for 3 replicas of tablet d769d9e140ad47baa3008973f5d5f337
I20260926 04:08:07.676721 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet d769d9e140ad47baa3008973f5d5f337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42832
I20260926 04:08:07.676831 8925 tablet_replica.cc:330] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9: stopping tablet replica
I20260926 04:08:07.676821 8625 tablet_service.cc:1514] Processing DeleteTablet for tablet d769d9e140ad47baa3008973f5d5f337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:37630
I20260926 04:08:07.676887 8925 raft_consensus.cc:2229] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:07.676919 8926 tablet_replica.cc:330] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: stopping tablet replica
I20260926 04:08:07.676927 8925 raft_consensus.cc:2258] T d769d9e140ad47baa3008973f5d5f337 P 1f833f6dbcf94eec981d0e950593b4b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.676936 8765 tablet_service.cc:1514] Processing DeleteTablet for tablet d769d9e140ad47baa3008973f5d5f337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:07 UTC) from {username='oryx', principal='oryx/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:39044
I20260926 04:08:07.676970 8926 raft_consensus.cc:2229] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:07.677022 8927 tablet_replica.cc:330] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: stopping tablet replica
I20260926 04:08:07.677059 8926 raft_consensus.cc:2258] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.677069 8927 raft_consensus.cc:2229] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:07.677099 8927 raft_consensus.cc:2258] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:07.677501 8927 ts_tablet_manager.cc:1905] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.677690 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 8420
I20260926 04:08:07.677781 8926 ts_tablet_manager.cc:1905] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:07.679404 8926 ts_tablet_manager.cc:1918] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:08:07.679461 8926 log.cc:1196] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/d769d9e140ad47baa3008973f5d5f337
I20260926 04:08:07.679500 8927 ts_tablet_manager.cc:1918] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 04:08:07.679554 8927 log.cc:1196] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/d769d9e140ad47baa3008973f5d5f337
I20260926 04:08:07.679689 8926 ts_tablet_manager.cc:1939] T d769d9e140ad47baa3008973f5d5f337 P fb912d2079654ebe8f0e77ca68f7c4cf: Deleting consensus metadata
I20260926 04:08:07.679798 8927 ts_tablet_manager.cc:1939] T d769d9e140ad47baa3008973f5d5f337 P d057124d1d6e4bb1bebb81000776b0cd: Deleting consensus metadata
I20260926 04:08:07.680027 8334 catalog_manager.cc:4841] TS fb912d2079654ebe8f0e77ca68f7c4cf (127.0.213.194:34569): tablet d769d9e140ad47baa3008973f5d5f337 (table test-table [id=8039c0153b1a4a4da8ca6e9086cc8684]) successfully deleted
I20260926 04:08:07.681499 8332 catalog_manager.cc:4841] TS d057124d1d6e4bb1bebb81000776b0cd (127.0.213.195:33521): tablet d769d9e140ad47baa3008973f5d5f337 (table test-table [id=8039c0153b1a4a4da8ca6e9086cc8684]) successfully deleted
W20260926 04:08:07.685532 8334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:44941 (error 108)
I20260926 04:08:07.685818 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 8560
I20260926 04:08:07.686112 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:07.685932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44941 (local address 127.0.0.1:42842)
0926 04:08:07.686032 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:07.686068 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:44941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":44,"threads_started":1}
W20260926 04:08:07.686196 8334 catalog_manager.cc:4584] TS 1f833f6dbcf94eec981d0e950593b4b9 (127.0.213.193:44941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d769d9e140ad47baa3008973f5d5f337: Network error: Client connection negotiation failed: client connection to 127.0.213.193:44941: connect: Connection refused (error 111)
I20260926 04:08:07.693054 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 8700
I20260926 04:08:07.699203 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 8316
2026-09-26T04:08:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3173 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8964](info): set up 2 sockets
Sep 26 04:08:07 dist-test-slave-xf24 krb5kdc[8964](info): commencing operation
krb5kdc: starting...
W20260926 04:08:08.756734 855 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:08:08 dist-test-slave-xf24 krb5kdc[8964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:08:08.880751 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40407
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:08.965324 8977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:08.965533 8977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:08.965555 8977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:08.966753 8977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:08.966800 8977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:08.966815 8977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:08.966828 8977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:08.966840 8977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:08.968437 8977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:08.968715 8977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:08.968984 8977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:08.973953 8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:08.973961 8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:08.974036 8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:08.974071 8977 server_base.cc:1034] running on GCE node
I20260926 04:08:08.974495 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:08.974753 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:08.975900 8977 hybrid_clock.cc:648] HybridClock initialized: now 1790395688975874 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T04:08:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1278 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:08:09 dist-test-slave-xf24 krb5kdc[8990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:08:09 dist-test-slave-xf24 krb5kdc[8990](info): set up 2 sockets
Sep 26 04:08:09 dist-test-slave-xf24 krb5kdc[8990](info): commencing operation
krb5kdc: starting...
W20260926 04:08:10.032784 855 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:08:10.160388 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41603
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:10.241369 9003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:10.241586 9003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:10.241608 9003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:10.242790 9003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:10.242836 9003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:10.242851 9003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:10.242862 9003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:10.242874 9003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:10.244490 9003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:10.244755 9003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:10.244999 9003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:10.250105 9011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.250108 9009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:10.250327 9003 server_base.cc:1034] running on GCE node
W20260926 04:08:10.250334 9008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:10.250663 9003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:10.250964 9003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:10.252161 9003 hybrid_clock.cc:648] HybridClock initialized: now 1790395690252145 us; error 49 us; skew 500 ppm
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:10.254808 9003 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:08:10.255380 9003 webserver.cc:460] Webserver started at http://127.0.213.254:38807/ using document root <none> and password file <none>
I20260926 04:08:10.255610 9003 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:10.255677 9003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:10.255836 9003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:10.256978 9003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "523fe3277d424c0193fdab16a818b54f"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "62dd09bdca69876cc78b95e800447dc0"
server_key_iv: "a2412139c4a5873a4c9dfe272f1006e1"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.257351 9003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "523fe3277d424c0193fdab16a818b54f"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "62dd09bdca69876cc78b95e800447dc0"
server_key_iv: "a2412139c4a5873a4c9dfe272f1006e1"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.258714 9003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:08:10.259512 9018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.259690 9003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:10.259796 9003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "523fe3277d424c0193fdab16a818b54f"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "62dd09bdca69876cc78b95e800447dc0"
server_key_iv: "a2412139c4a5873a4c9dfe272f1006e1"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.259892 9003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:10.270350 9003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:10.272876 9003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:10.273051 9003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:10.277367 9003 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:41603
I20260926 04:08:10.277395 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:41603 every 8 connection(s)
I20260926 04:08:10.277952 9003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:10.278661 9071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:10.281049 9071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Bootstrap starting.
I20260926 04:08:10.281595 9071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:10.281829 9071 log.cc:824] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:10.282511 9071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: No bootstrap required, opened a new log
I20260926 04:08:10.283998 9071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:10.284211 9071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:10.284240 9071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523fe3277d424c0193fdab16a818b54f, State: Initialized, Role: FOLLOWER
I20260926 04:08:10.284349 9071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:10.284428 9071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:10.284464 9071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:10.284535 9071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:10.285441 9071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:10.285566 9071 leader_election.cc:302] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523fe3277d424c0193fdab16a818b54f; no voters:
I20260926 04:08:10.285758 9071 leader_election.cc:288] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:08:10.285784 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9003
I20260926 04:08:10.285820 9074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:10.285900 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:08:10.285980 9074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 LEADER]: Becoming Leader. State: Replica: 523fe3277d424c0193fdab16a818b54f, State: Running, Role: LEADER
I20260926 04:08:10.286034 9071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:10.286067 9074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:10.286134 855 external_mini_cluster.cc:1428] Setting key 48f72397e043ad46eda1bfc22a6e57ea
I20260926 04:08:10.286479 9075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523fe3277d424c0193fdab16a818b54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } } }
I20260926 04:08:10.286589 9075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:10.286480 9076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523fe3277d424c0193fdab16a818b54f. Latest consensus state: current_term: 1 leader_uuid: "523fe3277d424c0193fdab16a818b54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } } }
I20260926 04:08:10.286778 9076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:10.286916 9084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:10.287698 9084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:10.289914 9084 catalog_manager.cc:1347] Generated new cluster ID: 31ae20382cdb4728a61dde7d0e3bf4d6
I20260926 04:08:10.289983 9084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:10.298523 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.286788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.213.254:41603)
0926 04:08:10.287084 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:10.287094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:10.287209 (+ 115us) server_negotiation.cc:407] Connection header received
0926 04:08:10.287425 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:10.287435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:10.287528 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:10.287650 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:10.288677 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.289791 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.290858 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.291497 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.291600 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.294542 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:10.294550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:10.294550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:10.294570 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:10.296715 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.297133 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.297137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.297138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.297260 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.297473 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.297473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.297474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.297817 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:10.297979 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:10.298058 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 04:08:10.298219 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
I20260926 04:08:10.304605 9084 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:08:10.305212 9084 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:10.317183 9084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Generated new TSK 0
I20260926 04:08:10.317435 9084 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:08:10.347527 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41603
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:10.431520 9097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:10.431735 9097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:10.431759 9097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:10.433028 9097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:10.433076 9097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:10.433104 9097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:10.434566 9097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:10.434760 9097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:10.434947 9097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:10.439982 9102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.439987 9103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.439987 9105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:10.440270 9097 server_base.cc:1034] running on GCE node
I20260926 04:08:10.440642 9097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:10.440899 9097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:10.442039 9097 hybrid_clock.cc:648] HybridClock initialized: now 1790395690442020 us; error 41 us; skew 500 ppm
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:10.444667 9097 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:08:10.445099 9097 webserver.cc:460] Webserver started at http://127.0.213.193:42413/ using document root <none> and password file <none>
I20260926 04:08:10.445333 9097 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:10.445410 9097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:10.445564 9097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:10.446729 9097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "602cbfce72e44c67a1dba360d9cea222"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "9abb19cf0b7fae542c16132f29a58b8d"
server_key_iv: "df2fb91c969cea532ef490a6982785ce"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.447094 9097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "602cbfce72e44c67a1dba360d9cea222"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "9abb19cf0b7fae542c16132f29a58b8d"
server_key_iv: "df2fb91c969cea532ef490a6982785ce"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.448525 9097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:10.449262 9112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.449429 9097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:10.449532 9097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "602cbfce72e44c67a1dba360d9cea222"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "9abb19cf0b7fae542c16132f29a58b8d"
server_key_iv: "df2fb91c969cea532ef490a6982785ce"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.449605 9097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:10.468942 9097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:10.471019 9097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:10.471190 9097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:10.471438 9097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:10.471742 9097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:10.471796 9097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.471839 9097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:10.471863 9097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.477661 9097 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:39393
I20260926 04:08:10.477739 9225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:39393 every 8 connection(s)
I20260926 04:08:10.478062 9097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:10.482870 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9097
I20260926 04:08:10.482986 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:08:10.483188 855 external_mini_cluster.cc:1428] Setting key b09133e52155847e063c3905038fa1a7
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:10.488720 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.478827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:40249 (local address 127.0.213.254:41603)
0926 04:08:10.478895 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:10.478898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:10.479232 (+ 334us) server_negotiation.cc:407] Connection header received
0926 04:08:10.480138 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:10.480140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:10.480154 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:10.480218 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:10.480848 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.481352 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.482413 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.482670 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.482756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.485252 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:10.485254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:10.485254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:10.485266 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:10.487131 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.487596 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.487597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.487598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.487666 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.487959 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.487960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.487960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.488161 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:10.488239 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:10.488530 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 04:08:10.488618 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:08:10.488965 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.478807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:40249)
0926 04:08:10.478976 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:10.479019 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:10.479968 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:10.480277 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:10.480281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:10.480549 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:10.480740 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:10.480744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.481479 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.481480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:10.482317 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:10.482319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.482764 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.482765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:10.482886 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.483588 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:10.483597 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:10.485148 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:10.487233 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:10.487241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:10.487244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:10.487498 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:10.487736 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:10.487737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:10.487737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:10.487870 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:10.488231 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:10.488238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:10.488390 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 04:08:10.488511 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 04:08:10.488689 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260926 04:08:10.489745 9226 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41603
I20260926 04:08:10.489868 9226 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:10.490057 9226 heartbeater.cc:511] Master 127.0.213.254:41603 requested a full tablet report, sending...
I20260926 04:08:10.490644 9035 ts_manager.cc:194] Registered new tserver with Master: 602cbfce72e44c67a1dba360d9cea222 (127.0.213.193:39393)
I20260926 04:08:10.491276 9035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:40249
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:08:10.512413 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41603
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:10.594480 9231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:10.594709 9231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:10.594738 9231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:10.595871 9231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:10.595906 9231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:10.595930 9231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:10.597504 9231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:10.597707 9231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:10.597891 9231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:10.603062 9236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.603075 9237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.603060 9239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:10.603287 9231 server_base.cc:1034] running on GCE node
I20260926 04:08:10.603528 9231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:10.603773 9231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:10.604940 9231 hybrid_clock.cc:648] HybridClock initialized: now 1790395690604932 us; error 42 us; skew 500 ppm
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:10.607362 9231 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:08:10.607815 9231 webserver.cc:460] Webserver started at http://127.0.213.194:41271/ using document root <none> and password file <none>
I20260926 04:08:10.608093 9231 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:10.608172 9231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:10.608330 9231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:10.609434 9231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "09f40d6d573d439f8f4a846a3b5af757"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "2addd0a56a8e8311ec7cee15c8aa400b"
server_key_iv: "a7c396934aa1523815d78729f3d38636"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.609794 9231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "09f40d6d573d439f8f4a846a3b5af757"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "2addd0a56a8e8311ec7cee15c8aa400b"
server_key_iv: "a7c396934aa1523815d78729f3d38636"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.611193 9231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:10.612165 9246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.612438 9231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:10.612558 9231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "09f40d6d573d439f8f4a846a3b5af757"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "2addd0a56a8e8311ec7cee15c8aa400b"
server_key_iv: "a7c396934aa1523815d78729f3d38636"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.612658 9231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:10.622339 9231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:10.624626 9231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:10.624799 9231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:10.625038 9231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:10.625350 9231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:10.625412 9231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.625468 9231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:10.625497 9231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.631536 9231 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:42487
I20260926 04:08:10.631619 9359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:42487 every 8 connection(s)
I20260926 04:08:10.631953 9231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:08:10.637439 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9231
I20260926 04:08:10.637558 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:08:10.637763 855 external_mini_cluster.cc:1428] Setting key 00f7fa8f40a4a93bc656c43fe2806a21
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:10.642644 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.632766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:39197 (local address 127.0.213.254:41603)
0926 04:08:10.632829 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:10.632832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:10.633035 (+ 203us) server_negotiation.cc:407] Connection header received
0926 04:08:10.634037 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:10.634039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:10.634052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:10.634097 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:10.634732 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.635241 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.636400 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.636596 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.636671 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.639457 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:10.639458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:10.639459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:10.639469 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:10.641171 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.641641 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.641643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.641643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.641704 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.641963 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.641964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.641965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.642090 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:10.642169 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:10.642458 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 04:08:10.642547 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 04:08:10.642861 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.632748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:39197)
0926 04:08:10.632912 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:10.632947 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:10.633918 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:10.634166 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:10.634172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:10.634443 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:10.634632 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:10.634636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.635347 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.635348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:10.636293 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:10.636297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.636682 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.636683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:10.636788 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.637351 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:10.637363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:10.639350 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:10.641272 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:10.641287 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:10.641290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:10.641535 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:10.641773 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:10.641774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:10.641775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:10.641891 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:10.642160 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:10.642167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:10.642321 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 04:08:10.642438 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:08:10.642607 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260926 04:08:10.643322 9360 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41603
I20260926 04:08:10.643424 9360 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:10.643627 9360 heartbeater.cc:511] Master 127.0.213.254:41603 requested a full tablet report, sending...
I20260926 04:08:10.644047 9035 ts_manager.cc:194] Registered new tserver with Master: 09f40d6d573d439f8f4a846a3b5af757 (127.0.213.194:42487)
I20260926 04:08:10.644501 9035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:39197
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:08:10.667099 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41603
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:10.749827 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:10.750065 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:10.750090 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:10.751925 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:10.751979 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:10.752010 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:10.753772 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:10.754061 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:10.754323 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:10.759840 9373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.759840 9370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:10.760015 9371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:10.760337 9365 server_base.cc:1034] running on GCE node
I20260926 04:08:10.760567 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:10.760831 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:10.762005 9365 hybrid_clock.cc:648] HybridClock initialized: now 1790395690761990 us; error 29 us; skew 500 ppm
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:10.764930 9365 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:08:10.765434 9365 webserver.cc:460] Webserver started at http://127.0.213.195:39997/ using document root <none> and password file <none>
I20260926 04:08:10.765663 9365 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:10.765741 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:10.765895 9365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:10.767364 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "586cfeead7e5465e9585d1870bdc01e9"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "70a7568911795bd1b3024d7b9842f66d"
server_key_iv: "608b22895c49b769faffa0d4912e1e4e"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.767809 9365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "586cfeead7e5465e9585d1870bdc01e9"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "70a7568911795bd1b3024d7b9842f66d"
server_key_iv: "608b22895c49b769faffa0d4912e1e4e"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.769573 9365 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 04:08:10.770402 9380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.770686 9365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:10.770795 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "586cfeead7e5465e9585d1870bdc01e9"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "70a7568911795bd1b3024d7b9842f66d"
server_key_iv: "608b22895c49b769faffa0d4912e1e4e"
server_key_version: "encryptionkey@0"
I20260926 04:08:10.770884 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:10.784657 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:10.787294 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:10.787472 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:10.787731 9365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:10.788161 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:10.788228 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.788272 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:10.788300 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:10.794273 9365 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:34285
I20260926 04:08:10.794314 9493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:34285 every 8 connection(s)
I20260926 04:08:10.794668 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:08:10 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:10.802492 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9365
I20260926 04:08:10.802598 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:08:10.802809 855 external_mini_cluster.cc:1428] Setting key 5a8d7ca33b5371fb99286751b268dc47
I20260926 04:08:10.805333 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.795492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:37133 (local address 127.0.213.254:41603)
0926 04:08:10.795558 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:10.795561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:10.795752 (+ 191us) server_negotiation.cc:407] Connection header received
0926 04:08:10.796809 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:10.796811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:10.796823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:10.796865 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:10.797529 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.798020 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.799048 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.799260 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.799354 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.801884 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:10.801886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:10.801887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:10.801898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:10.803745 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.804336 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.804337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.804337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.804397 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:10.804674 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:10.804674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:10.804674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:10.804797 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:10.804850 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:10.805196 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 04:08:10.805268 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:08:10.805475 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:10.795412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:37133)
0926 04:08:10.795613 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:10.795662 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:10.796695 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:10.796951 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:10.796957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:10.797222 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:10.797421 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:10.797425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.798127 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.798128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:10.798949 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:10.798951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:10.799356 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:10.799357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:10.799492 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:10.800028 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:10.800042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:10.801764 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:10.803856 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:10.803864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:10.803866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:10.804211 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:10.804476 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:10.804477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:10.804477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:10.804597 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:10.804877 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:10.804883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:10.805015 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 04:08:10.805108 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 04:08:10.805240 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260926 04:08:10.805925 9494 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41603
I20260926 04:08:10.806025 9494 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:10.806170 9494 heartbeater.cc:511] Master 127.0.213.254:41603 requested a full tablet report, sending...
I20260926 04:08:10.806535 9035 ts_manager.cc:194] Registered new tserver with Master: 586cfeead7e5465e9585d1870bdc01e9 (127.0.213.195:34285)
I20260926 04:08:10.806927 9035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:37133
I20260926 04:08:10.815330 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab.
Entry for principal oryx/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab.
WARNING: no policy specified for oryx/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab.
Entry for principal oryx/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab.
WARNING: no policy specified for oryx/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab.
Entry for principal oryx/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab.
I20260926 04:08:10.899730 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9097
I20260926 04:08:10.904562 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9231
I20260926 04:08:10.909646 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9365
I20260926 04:08:10.914381 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9003
I20260926 04:08:10.919543 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41603
--webserver_interface=127.0.213.254
--webserver_port=38807
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:11.006616 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:11.006836 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:11.006860 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:11.008086 9503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:11.008132 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:11.008147 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:11.008160 9503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:11.008220 9503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:11.009786 9503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=38807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:11.010061 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:11.010315 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:11.015468 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.015450 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.015455 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:11.015635 9503 server_base.cc:1034] running on GCE node
I20260926 04:08:11.016098 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:11.016446 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:11.017635 9503 hybrid_clock.cc:648] HybridClock initialized: now 1790395691017600 us; error 53 us; skew 500 ppm
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:11.024278 9503 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:08:11.024801 9503 webserver.cc:460] Webserver started at http://127.0.213.254:38807/ using document root <none> and password file <none>
I20260926 04:08:11.025053 9503 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:11.025137 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:11.026582 9503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:11.027318 9518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.027590 9503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:11.027709 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "523fe3277d424c0193fdab16a818b54f"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "62dd09bdca69876cc78b95e800447dc0"
server_key_iv: "a2412139c4a5873a4c9dfe272f1006e1"
server_key_version: "encryptionkey@0"
I20260926 04:08:11.028246 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:11.043392 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:11.045854 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:11.046051 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:11.050137 9503 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:41603
I20260926 04:08:11.050173 9570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:41603 every 8 connection(s)
I20260926 04:08:11.050710 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:11.052011 9571 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:08:11.052706 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Bootstrap starting.
I20260926 04:08:11.054687 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9503
I20260926 04:08:11.055035 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:39393
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:41603
--webserver_port=42413
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:08:11.061575 9571 log.cc:824] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:11.063302 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:08:11.063613 9571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Bootstrap complete.
I20260926 04:08:11.065883 9571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:11.066139 9571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523fe3277d424c0193fdab16a818b54f, State: Initialized, Role: FOLLOWER
I20260926 04:08:11.066325 9571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:11.066411 9571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:11.066466 9571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:11.066516 9571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:08:11.067523 9571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:11.067646 9571 leader_election.cc:302] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523fe3277d424c0193fdab16a818b54f; no voters:
I20260926 04:08:11.067857 9571 leader_election.cc:288] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:08:11.067904 9575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:08:11.068017 9575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [term 2 LEADER]: Becoming Leader. State: Replica: 523fe3277d424c0193fdab16a818b54f, State: Running, Role: LEADER
I20260926 04:08:11.068125 9575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } }
I20260926 04:08:11.068161 9571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:11.068384 9577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "523fe3277d424c0193fdab16a818b54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } } }
I20260926 04:08:11.068382 9575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523fe3277d424c0193fdab16a818b54f. Latest consensus state: current_term: 2 leader_uuid: "523fe3277d424c0193fdab16a818b54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523fe3277d424c0193fdab16a818b54f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41603 } } }
I20260926 04:08:11.068519 9577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:11.068529 9575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:11.070039 9591 catalog_manager.cc:1259] Loaded cluster ID: 31ae20382cdb4728a61dde7d0e3bf4d6
I20260926 04:08:11.070107 9591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: loading cluster ID for follower catalog manager: success
I20260926 04:08:11.070909 9591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: acquiring CA information for follower catalog manager: success
I20260926 04:08:11.071226 9591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:08:11.071316 9580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:11.071460 9580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:11.071542 9580 catalog_manager.cc:1259] Loaded cluster ID: 31ae20382cdb4728a61dde7d0e3bf4d6
I20260926 04:08:11.071573 9580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:08:11.071731 9580 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:11.071825 9580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 523fe3277d424c0193fdab16a818b54f: Loaded TSK: 0
I20260926 04:08:11.071964 9580 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:08:11.155267 9573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:11.155546 9573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:11.155589 9573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:11.156895 9573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:11.156978 9573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:11.157027 9573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:11.158581 9573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.193.keytab
--principal=oryx/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:39393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=42413
--tserver_master_addrs=127.0.213.254:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:11.158860 9573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:11.159106 9573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:11.164286 9597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.164315 9598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.164489 9600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:11.164690 9573 server_base.cc:1034] running on GCE node
I20260926 04:08:11.164893 9573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:11.165145 9573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:11.166316 9573 hybrid_clock.cc:648] HybridClock initialized: now 1790395691166302 us; error 32 us; skew 500 ppm
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:11.169337 9573 init.cc:377] Logged in from keytab as oryx/127.0.213.193@KRBTEST.COM (short username oryx)
I20260926 04:08:11.169992 9573 webserver.cc:460] Webserver started at http://127.0.213.193:42413/ using document root <none> and password file <none>
I20260926 04:08:11.170240 9573 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:11.170332 9573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:11.171823 9573 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:11.172593 9607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.172813 9573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:08:11.172925 9573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "602cbfce72e44c67a1dba360d9cea222"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "9abb19cf0b7fae542c16132f29a58b8d"
server_key_iv: "df2fb91c969cea532ef490a6982785ce"
server_key_version: "encryptionkey@0"
I20260926 04:08:11.173316 9573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:11.184250 9573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:11.186575 9573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:11.186754 9573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:11.187005 9573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:11.187314 9573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:11.187369 9573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.187440 9573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:11.187472 9573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.193933 9573 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:39393
I20260926 04:08:11.193977 9720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:39393 every 8 connection(s)
I20260926 04:08:11.194351 9573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:11.200858 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9573
I20260926 04:08:11.201288 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:42487
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:41603
--webserver_port=41271
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.204452 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.195211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:45809)
0926 04:08:11.195382 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.195431 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.195897 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.196442 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.196445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.197009 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.197201 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.197206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.198851 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.198852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.199696 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.199699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.200537 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.200537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.200657 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.201307 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.201318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.204228 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
W20260926 04:08:11.204561 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.195279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:45809 (local address 127.0.213.254:41603)
0926 04:08:11.195479 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.195488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.195524 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:08:11.196132 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.196140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.196208 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.196365 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.197336 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.198740 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.199793 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.200445 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.200562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.204364 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:45809: BlockingRecv error: recv got EOF from 127.0.213.193:45809 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
W20260926 04:08:11.204756 9721 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:41603 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.213.254:41603: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.209450 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.205006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:48159)
0926 04:08:11.205049 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.205060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.205101 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.205245 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.205245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.205409 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.205470 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.205473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.206099 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.206100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.206610 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.206612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.207017 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.207018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.207080 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.207541 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.207546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.209355 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 04:08:11.209501 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.205044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:48159 (local address 127.0.213.254:41603)
0926 04:08:11.205078 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.205080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.205087 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:11.205153 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.205154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.205165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.205242 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.205544 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.206017 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.206710 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.206912 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.207003 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.209434 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:48159: BlockingRecv error: recv got EOF from 127.0.213.193:48159 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.214258 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.210034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:41533)
0926 04:08:11.210122 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.210133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.210177 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.210310 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.210311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.210441 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.210469 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.210470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.211088 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.211089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.211673 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.211675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.212016 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.212017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.212091 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.212541 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.212546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.214172 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 04:08:11.214301 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.210087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:41533 (local address 127.0.213.254:41603)
0926 04:08:11.210124 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.210126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.210163 (+ 37us) server_negotiation.cc:407] Connection header received
0926 04:08:11.210238 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.210238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.210249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.210305 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.210545 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.211008 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.211749 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.211935 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.212016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.214239 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:41533: BlockingRecv error: recv got EOF from 127.0.213.193:41533 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:08:11.214638 9721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:08:11.292448 9725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:11.292668 9725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:11.292694 9725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:11.293834 9725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:11.293874 9725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:11.293902 9725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:11.295329 9725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.194.keytab
--principal=oryx/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=41271
--tserver_master_addrs=127.0.213.254:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:11.295526 9725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:11.295711 9725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:08:11.301215 9725 server_base.cc:1034] running on GCE node
W20260926 04:08:11.301179 9733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.301172 9731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.301172 9730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:11.301637 9725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:11.301949 9725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:11.303115 9725 hybrid_clock.cc:648] HybridClock initialized: now 1790395691303092 us; error 39 us; skew 500 ppm
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:11.305753 9725 init.cc:377] Logged in from keytab as oryx/127.0.213.194@KRBTEST.COM (short username oryx)
I20260926 04:08:11.306392 9725 webserver.cc:460] Webserver started at http://127.0.213.194:41271/ using document root <none> and password file <none>
I20260926 04:08:11.306622 9725 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:11.306701 9725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:11.308148 9725 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:11.308794 9740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.308961 9725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:11.309058 9725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "09f40d6d573d439f8f4a846a3b5af757"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "2addd0a56a8e8311ec7cee15c8aa400b"
server_key_iv: "a7c396934aa1523815d78729f3d38636"
server_key_version: "encryptionkey@0"
I20260926 04:08:11.309455 9725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:11.318439 9725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:11.320881 9725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:11.321071 9725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:11.321333 9725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:11.321666 9725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:11.321722 9725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.321761 9725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:11.321807 9725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.328361 9725 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:42487
I20260926 04:08:11.328408 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:42487 every 8 connection(s)
I20260926 04:08:11.328797 9725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 04:08:11.336736 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.329635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:51437 (local address 127.0.213.254:41603)
0926 04:08:11.329684 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.329686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.329901 (+ 215us) server_negotiation.cc:407] Connection header received
0926 04:08:11.330406 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.330407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.330423 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.330490 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.331657 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.332360 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.333429 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.333611 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.333669 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.336677 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:51437: BlockingRecv error: recv got EOF from 127.0.213.194:51437 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:11.336812 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.329569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:51437)
0926 04:08:11.329742 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.329785 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.330271 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.330568 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.330572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.331333 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.331549 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.331554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.332469 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.332470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.333339 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.333341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.333717 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.333718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.333822 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.334329 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.334339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.336601 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
W20260926 04:08:11.337064 9854 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:41603 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.213.254:41603: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
I20260926 04:08:11.337678 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9725
I20260926 04:08:11.338032 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:34285
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:41603
--webserver_port=39997
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.342716 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.337387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:55943)
0926 04:08:11.337426 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.337457 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.337502 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.337642 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.337643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.337803 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.337874 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.337876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.338666 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.338667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.339527 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.339529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.340029 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.340030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.340158 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.340630 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.340637 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.342627 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 04:08:11.342784 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.337423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:55943 (local address 127.0.213.254:41603)
0926 04:08:11.337479 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.337481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.337491 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:11.337570 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.337571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.337579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.337633 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.337991 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.338585 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.339615 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.339936 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.340015 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.342709 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:55943: BlockingRecv error: recv got EOF from 127.0.213.194:55943 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.347427 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.342992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:59201)
0926 04:08:11.343144 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.343153 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.343214 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.343371 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.343372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.343505 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.343535 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.343536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.344266 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.344267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.344764 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.344766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.345148 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.345148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.345203 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.345699 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.345703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.347338 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
W20260926 04:08:11.347473 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.343039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:59201 (local address 127.0.213.254:41603)
0926 04:08:11.343076 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.343078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.343201 (+ 123us) server_negotiation.cc:407] Connection header received
0926 04:08:11.343275 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.343276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.343295 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.343368 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.343612 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.344168 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.344868 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.345062 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.345147 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.347410 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:59201: BlockingRecv error: recv got EOF from 127.0.213.194:59201 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 04:08:11.347589 9854 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:08:11.420893 9857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:11.421103 9857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:11.421128 9857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:11.422322 9857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:11.422366 9857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:11.422394 9857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:11.423842 9857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/krb5kdc/oryx_127.0.213.195.keytab
--principal=oryx/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:34285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=39997
--tserver_master_addrs=127.0.213.254:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:11.424111 9857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:11.424357 9857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:11.429494 9864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:11.429587 9863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:11.429705 9857 server_base.cc:1034] running on GCE node
W20260926 04:08:11.429594 9866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:11.429983 9857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:11.430210 9857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:11.431401 9857 hybrid_clock.cc:648] HybridClock initialized: now 1790395691431379 us; error 44 us; skew 500 ppm
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:11.434165 9857 init.cc:377] Logged in from keytab as oryx/127.0.213.195@KRBTEST.COM (short username oryx)
I20260926 04:08:11.434593 9857 webserver.cc:460] Webserver started at http://127.0.213.195:39997/ using document root <none> and password file <none>
I20260926 04:08:11.434851 9857 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:11.434935 9857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:11.436352 9857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:11.436961 9873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.437145 9857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:11.437250 9857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "586cfeead7e5465e9585d1870bdc01e9"
format_stamp: "Formatted at 2026-09-26 04:08:10 on dist-test-slave-xf24"
server_key: "70a7568911795bd1b3024d7b9842f66d"
server_key_iv: "608b22895c49b769faffa0d4912e1e4e"
server_key_version: "encryptionkey@0"
I20260926 04:08:11.437616 9857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:11.449721 9857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:11.451828 9857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:11.451992 9857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:11.452247 9857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:11.452570 9857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:11.452625 9857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.452666 9857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:11.452695 9857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:11.458477 9857 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:34285
I20260926 04:08:11.458523 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:34285 every 8 connection(s)
I20260926 04:08:11.458863 9857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:08:11.463824 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 9857
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.466439 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.459615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:48359)
0926 04:08:11.459766 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.459806 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.460352 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.460623 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.460626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.461199 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.461394 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.461398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.462049 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.462050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.462847 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.462849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.463184 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.463185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.463279 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.463809 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.463818 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.466296 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260926 04:08:11.466452 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.459610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:48359 (local address 127.0.213.254:41603)
0926 04:08:11.459682 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.459684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.459893 (+ 209us) server_negotiation.cc:407] Connection header received
0926 04:08:11.460476 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.460477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.460488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.460552 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.461486 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.461969 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.462928 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.463109 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.463176 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.466389 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:48359: BlockingRecv error: recv got EOF from 127.0.213.195:48359 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 04:08:11.466701 9987 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:41603 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.213.254:41603: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.471362 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.466930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:55227)
0926 04:08:11.466961 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.466972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.467009 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.467165 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.467166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.467294 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.467333 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.467334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.468025 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.468026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.468582 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.468584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.469044 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.469045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.469126 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.469438 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.469443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.471281 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 04:08:11.471411 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.466992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:55227 (local address 127.0.213.254:41603)
0926 04:08:11.467052 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.467053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.467061 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:11.467079 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.467080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.467087 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.467160 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.467408 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.467952 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.468683 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.468973 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.469038 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.471370 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:55227: BlockingRecv error: recv got EOF from 127.0.213.195:55227 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:08:11.472695 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.464369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.0.213.254:41603)
0926 04:08:11.464498 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.464500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.464510 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:11.464533 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.464533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.464542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.464614 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.465267 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.465893 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.466840 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.467099 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.467175 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.468658 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:11.468662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:11.468663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:11.468674 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:11.470713 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:11.471487 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:11.471491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:11.471492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:11.471624 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:11.471883 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:11.471884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:11.471884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:11.472253 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:11.472376 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:11.472505 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 04:08:11.472590 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:11 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:11.476053 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:11.471710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:46433)
0926 04:08:11.471766 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:11.471777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:11.471812 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:11.471980 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:11.471980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:11.472126 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:11.472153 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.472155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.472930 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.472931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.473654 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:11.473656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.474028 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.474029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:11.474082 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.474413 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:11.474417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:11.475973 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:08:11.476116 9724 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:11.471810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:46433 (local address 127.0.213.254:41603)
0926 04:08:11.471871 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:11.471873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:11.471880 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:11.471900 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:11.471901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:11.471909 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:11.471966 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:11.472223 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.472838 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.473728 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:11.473926 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:11.474027 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:11.476035 (+ 2008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:46433: BlockingRecv error: recv got EOF from 127.0.213.195:46433 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 04:08:11.476274 9987 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 04:08:12 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:12 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:12.220245 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:12.215174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:32821)
0926 04:08:12.215330 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:12.215342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:12.215396 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:12.215557 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:12.215558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:12.215722 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:12.215781 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:12.215782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.216508 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.216509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:12.217315 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:12.217319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.217747 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.217748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:12.217807 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:12.218253 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:12.218259 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:12.220176 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
W20260926 04:08:12.220367 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:12.215271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:32821 (local address 127.0.213.254:41603)
0926 04:08:12.215418 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:12.215420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:12.215431 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:08:12.215456 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:12.215457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:12.215468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:12.215542 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:12.215872 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.216402 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.217403 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.217656 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.217745 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:12.220292 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:32821: BlockingRecv error: recv got EOF from 127.0.213.193:32821 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
Sep 26 04:08:12 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:12 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:12.353500 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:12.348126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:52387)
0926 04:08:12.348330 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:12.348344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:12.348421 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:12.348571 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:12.348572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:12.348736 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:12.348785 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:12.348787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.349455 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.349456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:12.350034 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:12.350037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.350468 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.350470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:12.350523 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:12.351281 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:12.351287 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:12.353413 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
W20260926 04:08:12.353663 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:12.348302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:52387 (local address 127.0.213.254:41603)
0926 04:08:12.348381 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:12.348383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:12.348398 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:08:12.348484 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:12.348485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:12.348496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:12.348549 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:12.348853 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.349352 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.350142 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.350370 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.350432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:12.353534 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:52387: BlockingRecv error: recv got EOF from 127.0.213.194:52387 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 04:08:12 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:12 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:12.482681 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:12.476832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:58163)
0926 04:08:12.477065 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:12.477081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:12.477175 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:12.477416 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:12.477417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:12.477624 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:12.477693 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:12.477695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.478444 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.478446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:12.479035 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:12.479036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.479467 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.479469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:12.479553 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:12.480346 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:12.480351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:12.482594 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":119,"threads_started":1}
W20260926 04:08:12.482877 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:12.476901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:58163 (local address 127.0.213.254:41603)
0926 04:08:12.476995 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:12.476998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:12.477150 (+ 152us) server_negotiation.cc:407] Connection header received
0926 04:08:12.477281 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:12.477282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:12.477296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:12.477356 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:12.477812 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.478362 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.479140 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:12.479353 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:12.479451 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:12.482750 (+ 3299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:58163: BlockingRecv error: recv got EOF from 127.0.213.195:58163 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 04:08:13 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:13 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:13.226702 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:13.220826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:36103)
0926 04:08:13.221031 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:13.221044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:13.221105 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:13.221278 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:13.221279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:13.221495 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:13.221561 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:13.221563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.222457 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.222458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:13.223056 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:13.223058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.223533 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.223535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:13.223600 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:13.224340 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:13.224347 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:13.226608 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":133,"threads_started":1}
W20260926 04:08:13.226977 9997 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:13.220976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:36103 (local address 127.0.213.254:41603)
0926 04:08:13.221134 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:13.221136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:13.221144 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:13.221172 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:13.221172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:13.221184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:13.221259 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:13.221635 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.222192 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.223180 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.223423 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.223518 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:13.226765 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:36103: BlockingRecv error: recv got EOF from 127.0.213.193:36103 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
Sep 26 04:08:13 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:13 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:13.360898 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:13.354149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:58547)
0926 04:08:13.354439 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:13.354461 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:13.354592 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:13.354814 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:13.354815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:13.355023 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:13.355074 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:13.355076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.355897 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.355899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:13.356680 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:13.356687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.357203 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.357203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:13.357263 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:13.357930 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:13.357935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:13.360818 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
W20260926 04:08:13.361054 9997 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:13.354307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:58547 (local address 127.0.213.254:41603)
0926 04:08:13.354446 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:13.354450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:13.354563 (+ 113us) server_negotiation.cc:407] Connection header received
0926 04:08:13.354722 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:13.354723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:13.354737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:13.354805 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:13.355158 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.355774 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.356861 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.357112 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.357205 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:13.360953 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:58547: BlockingRecv error: recv got EOF from 127.0.213.194:58547 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 04:08:13 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:13 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:13.488282 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:13.483253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:34701)
0926 04:08:13.483439 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:13.483452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:13.483553 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:13.483731 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:13.483732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:13.483928 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:13.483983 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:13.483985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.484689 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.484690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:13.485238 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:13.485240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.485626 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.485627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:13.485694 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:13.486262 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:13.486267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:13.488183 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
W20260926 04:08:13.488337 9997 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:13.483459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34701 (local address 127.0.213.254:41603)
0926 04:08:13.483528 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:13.483530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:13.483539 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:13.483636 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:13.483637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:13.483654 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:13.483721 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:13.484089 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.484600 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.485328 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:13.485519 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:13.485617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:13.488263 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:34701: BlockingRecv error: recv got EOF from 127.0.213.195:34701 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 04:08:14 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:14 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:14.232707 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:14.227308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:51669)
0926 04:08:14.227484 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:14.227498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:14.227582 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:14.227745 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:14.227746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:14.227917 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:14.227974 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:14.227976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.228677 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.228677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:14.229361 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:14.229363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.229721 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.229722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:14.229762 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:14.230267 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:14.230272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:14.232592 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
W20260926 04:08:14.232753 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:14.227438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:51669 (local address 127.0.213.254:41603)
0926 04:08:14.227593 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:14.227595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:14.227604 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:14.227632 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:14.227632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:14.227661 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:14.227725 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:14.228044 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.228587 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.229440 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.229656 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.229712 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:14.232681 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:51669: BlockingRecv error: recv got EOF from 127.0.213.193:51669 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
Sep 26 04:08:14 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:14 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:14.366242 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:14.361524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:46591)
0926 04:08:14.361712 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:14.361726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:14.361808 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:14.361935 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:14.361936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:14.362115 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:14.362166 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:14.362167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.362779 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.362780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:14.363345 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:14.363347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.363681 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.363682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:14.363730 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:14.364229 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:14.364233 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:14.366171 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
W20260926 04:08:14.366310 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:14.361685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:46591 (local address 127.0.213.254:41603)
0926 04:08:14.361739 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:14.361741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:14.361788 (+ 47us) server_negotiation.cc:407] Connection header received
0926 04:08:14.361866 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:14.361867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:14.361877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:14.361928 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:14.362231 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.362709 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.363418 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.363595 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.363646 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:14.366263 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:46591: BlockingRecv error: recv got EOF from 127.0.213.194:46591 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 04:08:14 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:14 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:14.493571 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:14.488826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:39827)
0926 04:08:14.488967 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:14.488984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:14.489040 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:14.489180 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:14.489181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:14.489337 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:14.489385 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:14.489386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.490100 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.490102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:14.490643 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:14.490644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.490984 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.490985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:14.491022 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:14.491547 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:14.491552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:14.493501 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
W20260926 04:08:14.493642 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:14.488992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:39827 (local address 127.0.213.254:41603)
0926 04:08:14.489055 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:14.489057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:14.489073 (+ 16us) server_negotiation.cc:407] Connection header received
0926 04:08:14.489095 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:14.489095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:14.489106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:14.489162 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:14.489496 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.490016 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.490725 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:14.490912 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:14.490978 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:14.493594 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:39827: BlockingRecv error: recv got EOF from 127.0.213.195:39827 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 04:08:15 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:15 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:15.238333 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:15.233289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:45403)
0926 04:08:15.233433 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:15.233453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:15.233508 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:15.233657 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:15.233658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:15.233812 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:15.233863 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:15.233865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.234821 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.234822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:15.235444 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:15.235446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.235831 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.235832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:15.235894 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:15.236447 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:15.236453 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:15.238272 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20260926 04:08:15.238444 10005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:15.233369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:45403 (local address 127.0.213.254:41603)
0926 04:08:15.233516 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:15.233517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:15.233529 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:08:15.233561 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:15.233561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:15.233572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:15.233645 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:15.233935 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.234728 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.235527 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.235742 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.235800 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:15.238367 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:45403: BlockingRecv error: recv got EOF from 127.0.213.193:45403 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Sep 26 04:08:15 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:15 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:15.372684 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:15.366889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:57737)
0926 04:08:15.367055 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:15.367067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:15.367135 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:15.367391 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:15.367392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:15.367628 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:15.367687 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:15.367689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.368427 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.368428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:15.369191 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:15.369194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.369674 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.369675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:15.369722 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:15.370424 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:15.370429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:15.372613 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
W20260926 04:08:15.372876 10005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:15.366991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:57737 (local address 127.0.213.254:41603)
0926 04:08:15.367061 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:15.367063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:15.367237 (+ 174us) server_negotiation.cc:407] Connection header received
0926 04:08:15.367264 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:15.367265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:15.367277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:15.367377 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:15.367768 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.368344 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.369330 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.369561 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.369626 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:15.372795 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:57737: BlockingRecv error: recv got EOF from 127.0.213.194:57737 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 04:08:15 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:15 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:15.499063 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:15.494257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:53151)
0926 04:08:15.494383 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:15.494396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:15.494469 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:15.494602 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:15.494602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:15.494778 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:15.494824 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:15.494826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.495521 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.495523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:15.496083 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:15.496085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.496459 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.496460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:15.496525 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:15.497005 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:15.497009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:15.498933 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
W20260926 04:08:15.499063 10005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:15.494352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:53151 (local address 127.0.213.254:41603)
0926 04:08:15.494402 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:15.494404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:15.494470 (+ 66us) server_negotiation.cc:407] Connection header received
0926 04:08:15.494528 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:15.494529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:15.494543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:15.494597 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:15.494907 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.495426 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.496171 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:15.496382 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:15.496460 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:15.499006 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:53151: BlockingRecv error: recv got EOF from 127.0.213.195:53151 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 04:08:16 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:16 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:16.248158 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:16.239000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:44613)
0926 04:08:16.239153 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:16.239164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:16.239218 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:16.239415 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:16.239416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:16.239591 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:16.239637 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:16.239639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.240455 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.240456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:16.241150 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:16.241153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.241689 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.241690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:16.241766 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:16.245825 (+ 4059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:16.245831 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:16.248020 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260926 04:08:16.248227 10009 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:16.239082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:44613 (local address 127.0.213.254:41603)
0926 04:08:16.239235 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:16.239237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:16.239249 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:08:16.239275 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:16.239275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:16.239286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:16.239370 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:16.239732 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.240345 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.241287 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.241598 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.241669 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:16.248148 (+ 6479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:44613: BlockingRecv error: recv got EOF from 127.0.213.193:44613 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
Sep 26 04:08:16 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:16 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:16.378621 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:16.373291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:40513)
0926 04:08:16.373442 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:16.373463 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:16.373523 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:16.373728 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:16.373730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:16.373902 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:16.373952 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:16.373953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.374714 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.374715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:16.375496 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:16.375498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.375908 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.375909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:16.375953 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:16.376544 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:16.376549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:16.378560 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
W20260926 04:08:16.378763 10009 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:16.373468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:40513 (local address 127.0.213.254:41603)
0926 04:08:16.373540 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:16.373542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:16.373550 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:08:16.373578 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:16.373578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:16.373588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:16.373640 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:16.374039 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.374603 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.375556 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.375807 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.375864 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:16.378675 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:40513: BlockingRecv error: recv got EOF from 127.0.213.194:40513 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 04:08:16 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:16 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:16.504453 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:16.499520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:35427)
0926 04:08:16.499709 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:16.499722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:16.499815 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:16.499948 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:16.499949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:16.500157 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:16.500204 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:16.500206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.500879 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.500881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:16.501415 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:16.501417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.501775 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.501776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:16.501831 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:16.502367 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:16.502372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:16.504387 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
W20260926 04:08:16.504619 10009 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:16.499615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:35427 (local address 127.0.213.254:41603)
0926 04:08:16.499694 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:16.499697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:16.499794 (+ 97us) server_negotiation.cc:407] Connection header received
0926 04:08:16.499872 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:16.499873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:16.499887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:16.499932 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:16.500287 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.500784 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.501512 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:16.501701 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:16.501778 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:16.504547 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:35427: BlockingRecv error: recv got EOF from 127.0.213.195:35427 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 04:08:17 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:17 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:17.253803 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:17.248687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:38129)
0926 04:08:17.248824 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:17.248839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:17.248890 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:17.249055 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:17.249055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:17.249220 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:17.249285 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:17.249286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.250283 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.250284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:17.250872 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:17.250875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.251257 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.251258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:17.251319 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:17.251843 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:17.251848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:17.253685 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260926 04:08:17.253961 10013 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:17.248774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:38129 (local address 127.0.213.254:41603)
0926 04:08:17.248915 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:17.248916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:17.248925 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:17.248950 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:17.248951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:17.248960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:17.249036 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:17.249395 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.250209 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.250958 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.251164 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.251223 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:17.253866 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:38129: BlockingRecv error: recv got EOF from 127.0.213.193:38129 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
Sep 26 04:08:17 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:17 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:17.384275 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:17.379213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:53353)
0926 04:08:17.379388 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:17.379398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:17.379470 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:17.379612 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:17.379613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:17.379771 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:17.379820 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:17.379821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.380506 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.380507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:17.381061 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:17.381062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.381444 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.381445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:17.381488 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:17.382003 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:17.382007 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:17.384208 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
W20260926 04:08:17.384469 10013 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:17.379300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:53353 (local address 127.0.213.254:41603)
0926 04:08:17.379389 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:17.379391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:17.379477 (+ 86us) server_negotiation.cc:407] Connection header received
0926 04:08:17.379520 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:17.379521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:17.379531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:17.379592 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:17.379904 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.380423 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.381162 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.381363 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.381430 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:17.384382 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:53353: BlockingRecv error: recv got EOF from 127.0.213.194:53353 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 04:08:17 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:17 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:17.509621 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:17.504992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:59813)
0926 04:08:17.505145 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:17.505158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:17.505230 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:17.505363 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:17.505364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:17.505536 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:17.505582 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:17.505584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.506258 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.506259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:17.506810 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:17.506812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.507201 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.507202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:17.507263 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:17.507695 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:17.507699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:17.509524 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
W20260926 04:08:17.509658 10013 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:17.505071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:59813 (local address 127.0.213.254:41603)
0926 04:08:17.505149 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:17.505152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:17.505228 (+ 76us) server_negotiation.cc:407] Connection header received
0926 04:08:17.505300 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:17.505301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:17.505312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:17.505359 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:17.505663 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.506170 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.506899 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:17.507114 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:17.507199 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:17.509599 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:59813: BlockingRecv error: recv got EOF from 127.0.213.195:59813 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 04:08:18 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:18 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:18.259114 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:18.254285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:46515)
0926 04:08:18.254430 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:18.254441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:18.254489 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:18.254717 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:18.254719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:18.254894 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:18.254946 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:18.254948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.255664 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.255665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:18.256242 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:18.256244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.256594 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.256595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:18.256633 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:18.257240 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:18.257245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:18.259053 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
W20260926 04:08:18.259276 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:18.254370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:46515 (local address 127.0.213.254:41603)
0926 04:08:18.254507 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:18.254509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:18.254526 (+ 17us) server_negotiation.cc:407] Connection header received
0926 04:08:18.254548 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:18.254549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:18.254560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:18.254630 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:18.255035 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.255562 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.256318 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.256524 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.256582 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:18.259184 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:46515: BlockingRecv error: recv got EOF from 127.0.213.193:46515 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
Sep 26 04:08:18 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:18 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:18.389680 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:18.384757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:52061)
0926 04:08:18.384897 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:18.384908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:18.384970 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:18.385182 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:18.385183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:18.385355 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:18.385406 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:18.385408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.386124 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.386125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:18.386742 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:18.386744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.387112 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.387113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:18.387164 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:18.387658 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:18.387663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:18.389608 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260926 04:08:18.389851 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:18.384829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:52061 (local address 127.0.213.254:41603)
0926 04:08:18.384937 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:18.384939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:18.384952 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:08:18.385054 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:18.385054 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:18.385066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:18.385127 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:18.385494 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.386036 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.386842 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.387030 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.387096 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:18.389747 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:52061: BlockingRecv error: recv got EOF from 127.0.213.194:52061 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 04:08:18 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:18 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:18.514793 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:18.510188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:56365)
0926 04:08:18.510331 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:18.510349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:18.510408 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:18.510524 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:18.510525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:18.510688 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:18.510737 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:18.510739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.511413 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.511414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:18.511976 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:18.511977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.512362 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.512363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:18.512407 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:18.512953 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:18.512958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:18.514734 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260926 04:08:18.514895 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:18.510290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:56365 (local address 127.0.213.254:41603)
0926 04:08:18.510349 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:18.510351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:18.510407 (+ 56us) server_negotiation.cc:407] Connection header received
0926 04:08:18.510452 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:18.510452 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:18.510464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:18.510520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:18.510804 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.511299 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.512044 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:18.512292 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:18.512348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:18.514829 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:56365: BlockingRecv error: recv got EOF from 127.0.213.195:56365 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 04:08:19 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:19 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:19.265415 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:19.259699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:54825)
0926 04:08:19.259887 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:19.259932 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:19.260004 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:19.260177 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:19.260178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:19.260338 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:19.260385 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:19.260387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.261188 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.261189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:19.261742 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:19.261745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.262357 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.262359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:19.262435 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:19.263152 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:19.263157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:19.265348 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
W20260926 04:08:19.265533 10021 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:19.259815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:54825 (local address 127.0.213.254:41603)
0926 04:08:19.259932 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:19.259934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:19.260002 (+ 68us) server_negotiation.cc:407] Connection header received
0926 04:08:19.260092 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:19.260092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:19.260103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:19.260156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:19.260511 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.261072 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.261925 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.262220 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.262318 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:19.265459 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:54825: BlockingRecv error: recv got EOF from 127.0.213.193:54825 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
Sep 26 04:08:19 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:19 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:19.396212 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:19.390403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:50077)
0926 04:08:19.390671 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:19.390687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:19.390795 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:19.390950 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:19.390951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:19.391173 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:19.391220 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:19.391222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.392123 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.392124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:19.392666 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:19.392667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.393067 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.393068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:19.393122 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:19.393711 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:19.393716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:19.396138 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
W20260926 04:08:19.396333 10021 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:19.390540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:50077 (local address 127.0.213.254:41603)
0926 04:08:19.390663 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:19.390665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:19.390775 (+ 110us) server_negotiation.cc:407] Connection header received
0926 04:08:19.390872 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:19.390873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:19.390885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:19.390935 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:19.391309 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.391990 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.392758 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.392969 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.393065 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:19.396237 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:50077: BlockingRecv error: recv got EOF from 127.0.213.194:50077 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 04:08:19 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:19 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:19.523597 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:19.515397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:47297)
0926 04:08:19.515614 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:19.515626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:19.515721 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:19.515888 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:19.515889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:19.516141 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:19.516197 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:19.516199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.516896 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.516898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:19.517471 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:19.517473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.517897 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.517897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:19.517955 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:19.521567 (+ 3612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:19.521573 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:19.523487 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
W20260926 04:08:19.523653 10021 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:19.515538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:47297 (local address 127.0.213.254:41603)
0926 04:08:19.515611 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:19.515613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:19.515700 (+ 87us) server_negotiation.cc:407] Connection header received
0926 04:08:19.515784 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:19.515785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:19.515799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:19.515875 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:19.516276 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.516806 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.517611 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:19.517810 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:19.517896 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:19.523587 (+ 5691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:47297: BlockingRecv error: recv got EOF from 127.0.213.195:47297 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 04:08:20 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:20 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:20.271384 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:20.265939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:49933)
0926 04:08:20.266089 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:20.266099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:20.266153 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:20.266308 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:20.266309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:20.266533 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:20.266581 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:20.266583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.267259 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.267260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:20.267844 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:20.267846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.268395 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.268396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:20.268455 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:20.269046 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:20.269052 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:20.271316 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260926 04:08:20.271512 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:20.266062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:49933 (local address 127.0.213.254:41603)
0926 04:08:20.266185 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:20.266187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:20.266194 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:08:20.266215 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:20.266215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:20.266225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:20.266284 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:20.266672 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.267180 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.267947 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.268316 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.268381 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:20.271423 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:49933: BlockingRecv error: recv got EOF from 127.0.213.193:49933 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
Sep 26 04:08:20 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:20 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:20.401928 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:20.396812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:55401)
0926 04:08:20.396942 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:20.396953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:20.397023 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:20.397217 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:20.397218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:20.397459 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:20.397510 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:20.397512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.398220 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.398221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:20.398774 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:20.398776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.399123 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.399124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:20.399179 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:20.399744 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:20.399749 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:20.401824 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260926 04:08:20.401980 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:20.396855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:55401 (local address 127.0.213.254:41603)
0926 04:08:20.396920 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:20.396922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:20.397021 (+ 99us) server_negotiation.cc:407] Connection header received
0926 04:08:20.397101 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:20.397102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:20.397114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:20.397179 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:20.397603 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.398115 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.398851 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.399037 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.399095 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:20.401914 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:55401: BlockingRecv error: recv got EOF from 127.0.213.194:55401 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 04:08:20 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:20 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:20.529227 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:20.524334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:40849)
0926 04:08:20.524509 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:20.524520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:20.524590 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:20.524724 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:20.524724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:20.524884 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:20.524932 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:20.524934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.525581 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.525582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:20.526102 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:20.526104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.526471 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.526472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:20.526532 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:20.526980 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:20.526997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:20.529124 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
W20260926 04:08:20.529284 10025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:20.524444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:40849 (local address 127.0.213.254:41603)
0926 04:08:20.524518 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:20.524521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:20.524590 (+ 69us) server_negotiation.cc:407] Connection header received
0926 04:08:20.524638 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:20.524638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:20.524651 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:20.524712 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:20.524996 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.525508 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.526195 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:20.526375 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:20.526429 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:20.529222 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:40849: BlockingRecv error: recv got EOF from 127.0.213.195:40849 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 04:08:21 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:21 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:21.280452 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:21.271945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:40953)
0926 04:08:21.272264 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:21.272275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:21.272343 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:21.272520 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:21.272521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:21.272687 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:21.272736 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:21.272738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.273405 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.273406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:21.273936 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:21.273937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.274277 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.274278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:21.274325 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:21.278245 (+ 3920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:21.278251 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:21.280388 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":164,"threads_started":1}
W20260926 04:08:21.280601 10029 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:21.272046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:40953 (local address 127.0.213.254:41603)
0926 04:08:21.272185 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:21.272187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:21.272343 (+ 156us) server_negotiation.cc:407] Connection header received
0926 04:08:21.272392 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:21.272393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:21.272403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:21.272506 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:21.272815 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.273320 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.274013 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.274201 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.274257 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:21.280490 (+ 6233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:40953: BlockingRecv error: recv got EOF from 127.0.213.193:40953 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":46,"threads_started":1}
Sep 26 04:08:21 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:21 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:21.407464 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:21.402523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:49893)
0926 04:08:21.402677 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:21.402696 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:21.402784 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:21.402975 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:21.402976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:21.403160 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:21.403203 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:21.403205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.403885 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.403886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:21.404460 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:21.404462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.404866 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.404867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:21.404926 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:21.405455 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:21.405461 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:21.407385 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
W20260926 04:08:21.407598 10029 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:21.402682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:49893 (local address 127.0.213.254:41603)
0926 04:08:21.402760 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:21.402762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:21.402772 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:21.402848 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:21.402849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:21.402861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:21.402965 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:21.403284 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.403799 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.404549 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.404777 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.404858 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:21.407527 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:49893: BlockingRecv error: recv got EOF from 127.0.213.194:49893 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 04:08:21 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:21 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:21.534849 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:21.529804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:50125)
0926 04:08:21.529986 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:21.530002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:21.530077 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:21.530220 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:21.530221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:21.530390 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:21.530446 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:21.530447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.531175 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.531177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:21.531756 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:21.531758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.532174 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.532177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:21.532222 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:21.532725 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:21.532729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:21.534749 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
W20260926 04:08:21.534900 10029 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:21.529979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:50125 (local address 127.0.213.254:41603)
0926 04:08:21.530047 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:21.530049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:21.530062 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:08:21.530141 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:21.530142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:21.530154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:21.530196 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:21.530509 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.531040 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.531837 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:21.532017 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:21.532165 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:21.534829 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:50125: BlockingRecv error: recv got EOF from 127.0.213.195:50125 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 04:08:22 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:22 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:22.285979 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:22.281097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:33629)
0926 04:08:22.281245 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:22.281256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:22.281307 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:22.281519 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:22.281520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:22.281702 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:22.281757 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:22.281759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.282537 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.282538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:22.283113 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:22.283115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.283513 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.283514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:22.283571 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:22.284142 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:22.284149 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:22.285906 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260926 04:08:22.286206 10033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:22.281171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:33629 (local address 127.0.213.254:41603)
0926 04:08:22.281327 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:22.281329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:22.281366 (+ 37us) server_negotiation.cc:407] Connection header received
0926 04:08:22.281400 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:22.281401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:22.281412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:22.281473 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:22.281891 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.282453 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.283202 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.283440 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.283502 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:22.286138 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:33629: BlockingRecv error: recv got EOF from 127.0.213.193:33629 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
Sep 26 04:08:22 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:22 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:22.413664 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:22.408176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:39817)
0926 04:08:22.408299 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:22.408317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:22.408374 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:22.408576 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:22.408576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:22.408749 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:22.408797 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:22.408799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.409604 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.409605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:22.410359 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:22.410362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.410722 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.410723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:22.410764 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:22.411382 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:22.411387 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:22.413586 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260926 04:08:22.413741 10033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:22.408268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:39817 (local address 127.0.213.254:41603)
0926 04:08:22.408354 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:22.408356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:22.408375 (+ 19us) server_negotiation.cc:407] Connection header received
0926 04:08:22.408463 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:22.408464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:22.408475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:22.408525 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:22.408900 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.409487 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.410450 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.410646 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.410707 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:22.413691 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:39817: BlockingRecv error: recv got EOF from 127.0.213.194:39817 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 04:08:22 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:22 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:22.540462 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:22.535406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:49673)
0926 04:08:22.535733 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:22.535745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:22.535852 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:22.536007 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:22.536008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:22.536222 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:22.536269 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:22.536271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.536986 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.536987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:22.537550 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:22.537552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.537934 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.537935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:22.537998 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:22.538484 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:22.538489 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:22.540379 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":237,"threads_started":1}
W20260926 04:08:22.540542 10033 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:22.535538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:49673 (local address 127.0.213.254:41603)
0926 04:08:22.535608 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:22.535610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:22.535831 (+ 221us) server_negotiation.cc:407] Connection header received
0926 04:08:22.535934 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:22.535935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:22.535949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:22.536003 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:22.536364 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.536864 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.537645 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:22.537839 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:22.537932 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:22.540465 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:49673: BlockingRecv error: recv got EOF from 127.0.213.195:49673 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 04:08:23 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:23 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:23.291318 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:23.286553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:52751)
0926 04:08:23.286732 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:23.286743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:23.286817 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:23.286958 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:23.286959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:23.287129 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:23.287184 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:23.287186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.287894 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.287895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:23.288527 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:23.288529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.288899 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.288900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:23.288952 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:23.289413 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:23.289418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:23.291234 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
W20260926 04:08:23.291502 10037 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:23.286661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:52751 (local address 127.0.213.254:41603)
0926 04:08:23.286821 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:23.286823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:23.286851 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:08:23.286875 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:23.286876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:23.286885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:23.286936 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:23.287264 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.287809 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.288602 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.288790 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.288847 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:23.291369 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:52751: BlockingRecv error: recv got EOF from 127.0.213.193:52751 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
Sep 26 04:08:23 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:23 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:23.418978 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:23.414206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:40259)
0926 04:08:23.414444 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:23.414485 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:23.414570 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:23.414775 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:23.414775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:23.414986 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:23.415034 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:23.415035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.415712 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.415713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:23.416342 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:23.416344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.416671 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.416671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:23.416709 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:23.417145 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:23.417149 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:23.418905 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":174,"threads_started":1}
W20260926 04:08:23.419106 10037 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:23.414310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:40259 (local address 127.0.213.254:41603)
0926 04:08:23.414382 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:23.414384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:23.414559 (+ 175us) server_negotiation.cc:407] Connection header received
0926 04:08:23.414621 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:23.414622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:23.414635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:23.414714 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:23.415100 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.415603 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.416420 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.416604 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.416661 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:23.419018 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:40259: BlockingRecv error: recv got EOF from 127.0.213.194:40259 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 04:08:23 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:23 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:23.545799 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:23.540963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:43933)
0926 04:08:23.541097 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:23.541110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:23.541207 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:23.541410 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:23.541411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:23.541603 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:23.541649 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:23.541651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.542311 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.542312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:23.542848 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:23.542849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.543218 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.543219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:23.543283 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:23.543759 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:23.543764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:23.545701 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260926 04:08:23.545850 10037 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:23.541029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:43933 (local address 127.0.213.254:41603)
0926 04:08:23.541102 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:23.541104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:23.541187 (+ 83us) server_negotiation.cc:407] Connection header received
0926 04:08:23.541299 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:23.541300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:23.541314 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:23.541388 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:23.541731 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.542224 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.542935 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:23.543133 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:23.543208 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:23.545784 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:43933: BlockingRecv error: recv got EOF from 127.0.213.195:43933 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 04:08:24 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:24 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:24.296811 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:24.291877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:60245)
0926 04:08:24.292033 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:24.292045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:24.292156 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:24.292308 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:24.292309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:24.292481 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:24.292529 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:24.292531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.293239 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.293240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:24.293804 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:24.293806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.294196 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.294197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:24.294262 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:24.294771 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:24.294777 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:24.296718 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
W20260926 04:08:24.296936 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:24.291991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:60245 (local address 127.0.213.254:41603)
0926 04:08:24.292165 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:24.292167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:24.292176 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:24.292203 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:24.292203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:24.292214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:24.292293 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:24.292602 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.293159 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.293889 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.294105 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.294168 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:24.296837 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:60245: BlockingRecv error: recv got EOF from 127.0.213.193:60245 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
Sep 26 04:08:24 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:24 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:24.424335 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:24.419459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:33313)
0926 04:08:24.419639 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:24.419653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:24.419759 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:24.419944 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:24.419945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:24.420190 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:24.420269 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:24.420271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.420938 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.420939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:24.421475 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:24.421476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.421854 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.421855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:24.421916 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:24.422417 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:24.422422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:24.424239 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
W20260926 04:08:24.424371 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:24.419592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:33313 (local address 127.0.213.254:41603)
0926 04:08:24.419650 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:24.419653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:24.419738 (+ 85us) server_negotiation.cc:407] Connection header received
0926 04:08:24.419838 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:24.419839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:24.419852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:24.419924 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:24.420357 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.420858 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.421575 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.421779 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.421857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:24.424312 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:33313: BlockingRecv error: recv got EOF from 127.0.213.194:33313 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 04:08:24 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:24 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:24.551355 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:24.546365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:34813)
0926 04:08:24.546518 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:24.546530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:24.546599 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:24.546766 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:24.546766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:24.546934 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:24.546980 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:24.546981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.547616 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.547617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:24.548189 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:24.548191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.548751 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.548752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:24.548828 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:24.549330 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:24.549335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:24.551236 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
W20260926 04:08:24.551383 10041 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:24.546465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34813 (local address 127.0.213.254:41603)
0926 04:08:24.546525 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:24.546528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:24.546596 (+ 68us) server_negotiation.cc:407] Connection header received
0926 04:08:24.546655 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:24.546656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:24.546668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:24.546731 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:24.547047 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.547549 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.548250 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:24.548589 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:24.548684 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:24.551317 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:34813: BlockingRecv error: recv got EOF from 127.0.213.195:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 04:08:25 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:25 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:25.302389 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:25.297365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:45313)
0926 04:08:25.297525 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:25.297540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:25.297611 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:25.297783 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:25.297784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:25.297952 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:25.298015 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:25.298017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.298731 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.298733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:25.299259 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:25.299260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.299635 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.299636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:25.299700 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:25.300329 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:25.300334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:25.302291 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
W20260926 04:08:25.302462 10045 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:25.297426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:45313 (local address 127.0.213.254:41603)
0926 04:08:25.297611 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:25.297612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:25.297622 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:25.297687 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:25.297687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:25.297698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:25.297765 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:25.298103 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.298639 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.299351 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.299551 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.299630 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:25.302387 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:45313: BlockingRecv error: recv got EOF from 127.0.213.193:45313 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
Sep 26 04:08:25 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:25 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:25.429296 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:25.424885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:44197)
0926 04:08:25.425015 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:25.425026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:25.425085 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:25.425223 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:25.425224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:25.425409 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:25.425455 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:25.425457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.426068 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.426069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:25.426613 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:25.426614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.426947 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.426948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:25.426986 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:25.427459 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:25.427464 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:25.429224 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":51,"threads_started":1}
W20260926 04:08:25.429383 10045 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:25.424960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:44197 (local address 127.0.213.254:41603)
0926 04:08:25.425047 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:25.425049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:25.425073 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:08:25.425132 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:25.425140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:25.425150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:25.425209 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:25.425520 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.426002 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.426701 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.426883 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.426938 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:25.429318 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:44197: BlockingRecv error: recv got EOF from 127.0.213.194:44197 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 04:08:25 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:25 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.195@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:25.556558 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:25.551946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.195:39453)
0926 04:08:25.552132 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:25.552144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:25.552204 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:25.552341 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:25.552341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:25.552515 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:25.552561 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:25.552563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.553232 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.553233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:25.553780 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:25.553782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.554118 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.554119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:25.554158 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:25.554668 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:25.554672 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:25.556452 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
W20260926 04:08:25.556752 10045 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:25.552051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:39453 (local address 127.0.213.254:41603)
0926 04:08:25.552185 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:25.552187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:25.552196 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:08:25.552253 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:25.552254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:25.552268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:25.552319 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:25.552656 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.553152 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.553865 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:25.554048 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:25.554108 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:25.556607 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.195:39453: BlockingRecv error: recv got EOF from 127.0.213.195:39453 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 04:08:26 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:26 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.193@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:26.313804 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.303071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.193:60517)
0926 04:08:26.303273 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:26.303285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:26.303353 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:26.303531 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:26.303532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:26.303776 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:26.303825 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.303827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.304579 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.304580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.305145 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.305147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.305535 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.305536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.305585 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.311439 (+ 5854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:26.311448 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:26.313685 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
W20260926 04:08:26.313948 10049 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:26.303173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:60517 (local address 127.0.213.254:41603)
0926 04:08:26.303281 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:26.303283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:26.303338 (+ 55us) server_negotiation.cc:407] Connection header received
0926 04:08:26.303408 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:26.303409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:26.303422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:26.303491 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:26.303903 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.304438 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.305242 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.305435 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.305497 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.313850 (+ 8353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.193:60517: BlockingRecv error: recv got EOF from 127.0.213.193:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
Sep 26 04:08:26 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 04:08:26 dist-test-slave-xf24 krb5kdc[8990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.213.194@KRBTEST.COM for oryx/127.0.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 04:08:26.434908 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.429944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41603 (local address 127.0.213.194:36901)
0926 04:08:26.430104 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:26.430118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:26.430194 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:26.430357 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:26.430358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:26.430511 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:26.430556 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.430558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.431305 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.431306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.431929 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.431931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.432333 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.432334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.432392 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.432835 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:26.432840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:26.434799 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.213.254:41603: Server oryx/127.0.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
W20260926 04:08:26.434945 10049 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:26.430055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:36901 (local address 127.0.213.254:41603)
0926 04:08:26.430131 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:26.430133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:26.430194 (+ 61us) server_negotiation.cc:407] Connection header received
0926 04:08:26.430244 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:26.430245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:26.430257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:26.430314 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:26.430642 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.431210 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.432003 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.432234 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.432299 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.434884 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.213.194:36901: BlockingRecv error: recv got EOF from 127.0.213.194:36901 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:08:26.467654 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9573
I20260926 04:08:26.472146 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9725
I20260926 04:08:26.476382 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9857
I20260926 04:08:26.480736 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 9503
2026-09-26T04:08:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17502 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T04:08:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:26Z Disabled control of system clock
I20260926 04:08:26.508697 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:39993
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 04:08:26.589108 10054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:26.589321 10054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:26.589342 10054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:26.590462 10054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:26.590499 10054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:26.590512 10054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:26.590524 10054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:26.590536 10054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:26.591814 10054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:39993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:26.592003 10054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:26.592311 10054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:26.597426 10062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:26.597440 10059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:26.597464 10054 server_base.cc:1034] running on GCE node
W20260926 04:08:26.597450 10060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:26.597909 10054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:26.598210 10054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:26.599364 10054 hybrid_clock.cc:648] HybridClock initialized: now 1790395706599358 us; error 35 us; skew 500 ppm
I20260926 04:08:26.600740 10054 webserver.cc:460] Webserver started at http://127.0.213.254:46395/ using document root <none> and password file <none>
I20260926 04:08:26.600998 10054 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:26.601048 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:26.601224 10054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:26.602351 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "d06f36515f0c411bbf5f2135a44d2066"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "c9e8e59d65b61a075c54fbb852bd9388"
server_key_iv: "5fc29c280fc08f5665949e8301e0c33f"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.602716 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "d06f36515f0c411bbf5f2135a44d2066"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "c9e8e59d65b61a075c54fbb852bd9388"
server_key_iv: "5fc29c280fc08f5665949e8301e0c33f"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.604001 10054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:26.604815 10068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.604984 10054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:26.605057 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "d06f36515f0c411bbf5f2135a44d2066"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "c9e8e59d65b61a075c54fbb852bd9388"
server_key_iv: "5fc29c280fc08f5665949e8301e0c33f"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.605109 10054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:26.614115 10054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:26.614370 10054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:26.614471 10054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:26.618263 10054 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:39993
I20260926 04:08:26.618295 10120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:39993 every 8 connection(s)
I20260926 04:08:26.618629 10054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:26.619257 10121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:26.621861 10121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066: Bootstrap starting.
I20260926 04:08:26.622419 10121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:26.622655 10121 log.cc:824] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:26.622928 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10054
I20260926 04:08:26.623039 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:08:26.623257 855 external_mini_cluster.cc:1428] Setting key e3c2cfb74f9c302d767ed1927897b9a2
I20260926 04:08:26.623383 10121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066: No bootstrap required, opened a new log
I20260926 04:08:26.625603 10121 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06f36515f0c411bbf5f2135a44d2066" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 39993 } }
I20260926 04:08:26.625765 10121 raft_consensus.cc:374] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:26.625808 10121 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06f36515f0c411bbf5f2135a44d2066, State: Initialized, Role: FOLLOWER
I20260926 04:08:26.625941 10121 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06f36515f0c411bbf5f2135a44d2066" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 39993 } }
I20260926 04:08:26.626049 10121 raft_consensus.cc:388] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:26.626094 10121 raft_consensus.cc:482] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:26.626143 10121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:26.626714 10121 raft_consensus.cc:504] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06f36515f0c411bbf5f2135a44d2066" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 39993 } }
I20260926 04:08:26.626837 10121 leader_election.cc:302] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d06f36515f0c411bbf5f2135a44d2066; no voters:
I20260926 04:08:26.627018 10121 leader_election.cc:288] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:08:26.627055 10126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:26.627177 10126 raft_consensus.cc:686] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [term 1 LEADER]: Becoming Leader. State: Replica: d06f36515f0c411bbf5f2135a44d2066, State: Running, Role: LEADER
I20260926 04:08:26.627314 10121 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:26.627317 10126 consensus_queue.cc:237] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06f36515f0c411bbf5f2135a44d2066" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 39993 } }
I20260926 04:08:26.627648 10126 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06f36515f0c411bbf5f2135a44d2066. Latest consensus state: current_term: 1 leader_uuid: "d06f36515f0c411bbf5f2135a44d2066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06f36515f0c411bbf5f2135a44d2066" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 39993 } } }
I20260926 04:08:26.627682 10128 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d06f36515f0c411bbf5f2135a44d2066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06f36515f0c411bbf5f2135a44d2066" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 39993 } } }
I20260926 04:08:26.627821 10126 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:26.627835 10128 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066 [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:26.628103 10134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:26.628672 10134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:26.630291 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.623799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.0.213.254:39993)
0926 04:08:26.624003 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:26.624011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:26.624036 (+ 25us) server_negotiation.cc:407] Connection header received
0926 04:08:26.624772 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:26.624780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:26.624840 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:26.624949 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:26.625212 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.626727 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.627809 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.628468 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.628565 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.629541 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:26.629544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:26.629545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:26.629746 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:26.629788 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:26.629811 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:26.629879 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:26.629998 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 04:08:26.630091 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260926 04:08:26.630714 10134 catalog_manager.cc:1347] Generated new cluster ID: 089f79f80b3d43d6957296a7befd9652
I20260926 04:08:26.630762 10134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:08:26.644488 10134 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:08:26.645239 10134 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:26.649370 10134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d06f36515f0c411bbf5f2135a44d2066: Generated new TSK 0
I20260926 04:08:26.649564 10134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:08:26.657783 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:39993
--builtin_ntp_servers=127.0.213.212:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 04:08:26.739939 10145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:26.740198 10145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:26.740236 10145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:26.741422 10145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:26.741490 10145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:26.741531 10145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:26.742965 10145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:39993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:26.743242 10145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:26.743546 10145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:26.748588 10150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:26.748790 10145 server_base.cc:1034] running on GCE node
W20260926 04:08:26.748592 10153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:26.748617 10151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:26.749195 10145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:26.749495 10145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:26.750687 10145 hybrid_clock.cc:648] HybridClock initialized: now 1790395706750670 us; error 34 us; skew 500 ppm
I20260926 04:08:26.752178 10145 webserver.cc:460] Webserver started at http://127.0.213.193:34813/ using document root <none> and password file <none>
I20260926 04:08:26.752465 10145 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:26.752538 10145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:26.752693 10145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:26.753787 10145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "800cbdeb5cf54539ad2b8073750c75a8"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "0f3014469afbbdfcab813403258060be"
server_key_iv: "191f6fd0e34f3213ff613cde8a3cb00e"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.754163 10145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "800cbdeb5cf54539ad2b8073750c75a8"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "0f3014469afbbdfcab813403258060be"
server_key_iv: "191f6fd0e34f3213ff613cde8a3cb00e"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.755537 10145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:26.756412 10159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.756733 10145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:26.756860 10145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "800cbdeb5cf54539ad2b8073750c75a8"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "0f3014469afbbdfcab813403258060be"
server_key_iv: "191f6fd0e34f3213ff613cde8a3cb00e"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.756944 10145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:26.775347 10145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:26.775722 10145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:26.775902 10145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:26.776278 10145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:26.776664 10145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:26.776727 10145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.776774 10145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:26.776801 10145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.783380 10145 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:43575
I20260926 04:08:26.783455 10272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:43575 every 8 connection(s)
I20260926 04:08:26.783834 10145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:26.789014 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.784691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:56739 (local address 127.0.213.254:39993)
0926 04:08:26.784766 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:26.784773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:26.785034 (+ 261us) server_negotiation.cc:407] Connection header received
0926 04:08:26.785519 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:26.785520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:26.785536 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:26.785612 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:26.786253 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.786765 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.787721 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.787906 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.787965 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.788770 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:26.788771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:26.788771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:26.788775 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:26.788791 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:26.788795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:26.788849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:26.788929 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 04:08:26.788951 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:08:26.789148 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.784683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:39993 (local address 127.0.213.193:56739)
0926 04:08:26.784902 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:26.784944 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:26.785427 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:26.785665 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:26.785668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:26.785677 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:26.786156 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.786160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.786851 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.786851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.787643 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.787645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.787985 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.787986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.788114 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.788623 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:08:26.788623 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:26.788651 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:08:26.788673 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:26.788858 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:26.788861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:26.788864 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:08:26.788916 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:08:26.788992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260926 04:08:26.789701 10273 heartbeater.cc:347] Connected to a master server at 127.0.213.254:39993
I20260926 04:08:26.789791 10273 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:26.789940 10273 heartbeater.cc:511] Master 127.0.213.254:39993 requested a full tablet report, sending...
I20260926 04:08:26.790437 10085 ts_manager.cc:194] Registered new tserver with Master: 800cbdeb5cf54539ad2b8073750c75a8 (127.0.213.193:43575)
I20260926 04:08:26.791088 10085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.193:56739
I20260926 04:08:26.793059 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10145
I20260926 04:08:26.793164 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:08:26.793357 855 external_mini_cluster.cc:1428] Setting key 251a3e6cb0d197d681ab1e290faa4a94
I20260926 04:08:26.794395 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:39993
--builtin_ntp_servers=127.0.213.212:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 04:08:26.876417 10276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:26.876669 10276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:26.876705 10276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:26.877874 10276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:26.877947 10276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:26.877986 10276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:26.879335 10276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:39993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:26.879602 10276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:26.880025 10276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:26.884969 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:26.884956 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:26.884958 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:26.885047 10276 server_base.cc:1034] running on GCE node
I20260926 04:08:26.885455 10276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:26.885711 10276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:26.886885 10276 hybrid_clock.cc:648] HybridClock initialized: now 1790395706886874 us; error 46 us; skew 500 ppm
I20260926 04:08:26.888185 10276 webserver.cc:460] Webserver started at http://127.0.213.194:40271/ using document root <none> and password file <none>
I20260926 04:08:26.888435 10276 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:26.888504 10276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:26.888702 10276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:26.889807 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "e1a9991ceb264e5e9f39cbdb3f00f881"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "581d87460d5fb24b50ae4e41792054e2"
server_key_iv: "efa1f12fd46d4ba01a69c8a6976d5ca9"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.890183 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "e1a9991ceb264e5e9f39cbdb3f00f881"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "581d87460d5fb24b50ae4e41792054e2"
server_key_iv: "efa1f12fd46d4ba01a69c8a6976d5ca9"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.891566 10276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:08:26.892379 10290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.892560 10276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:26.892657 10276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "e1a9991ceb264e5e9f39cbdb3f00f881"
format_stamp: "Formatted at 2026-09-26 04:08:26 on dist-test-slave-xf24"
server_key: "581d87460d5fb24b50ae4e41792054e2"
server_key_iv: "efa1f12fd46d4ba01a69c8a6976d5ca9"
server_key_version: "encryptionkey@0"
I20260926 04:08:26.892748 10276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:26.914804 10276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:26.915129 10276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:26.915292 10276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:26.915539 10276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:26.915880 10276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:26.915934 10276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.915969 10276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:26.915998 10276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:26.922222 10276 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:43543
I20260926 04:08:26.922277 10403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:43543 every 8 connection(s)
I20260926 04:08:26.922638 10276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:08:26.928345 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.923413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:53377 (local address 127.0.213.254:39993)
0926 04:08:26.923472 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:26.923474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:26.923731 (+ 257us) server_negotiation.cc:407] Connection header received
0926 04:08:26.924438 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:26.924439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:26.924452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:26.924499 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:26.925200 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.925709 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.926720 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.926922 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.927023 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.928039 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:26.928041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:26.928041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:26.928046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:26.928077 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:26.928082 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:26.928126 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:26.928213 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 04:08:26.928245 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 04:08:26.928699 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:26.923408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:39993 (local address 127.0.213.194:53377)
0926 04:08:26.923604 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:26.923644 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:26.924320 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:26.924592 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:26.924596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:26.924605 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:26.925100 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.925104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.925811 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.925812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.926628 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:26.926630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:26.927006 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:26.927007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:26.927173 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:26.927890 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:08:26.927890 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:26.927921 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:08:26.927944 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:26.928144 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:26.928149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:26.928152 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:08:26.928216 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:08:26.928325 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260926 04:08:26.929388 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10276
I20260926 04:08:26.929482 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:08:26.929684 855 external_mini_cluster.cc:1428] Setting key 7237ad6c277598617a84646b530a7ec8
I20260926 04:08:26.930140 10404 heartbeater.cc:347] Connected to a master server at 127.0.213.254:39993
I20260926 04:08:26.930236 10404 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:26.930420 10404 heartbeater.cc:511] Master 127.0.213.254:39993 requested a full tablet report, sending...
I20260926 04:08:26.930848 10085 ts_manager.cc:194] Registered new tserver with Master: e1a9991ceb264e5e9f39cbdb3f00f881 (127.0.213.194:43543)
I20260926 04:08:26.931190 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:39993
--builtin_ntp_servers=127.0.213.212:34327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 04:08:26.931354 10085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.194:53377
W20260926 04:08:27.014194 10407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:27.014413 10407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:27.014436 10407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:27.015609 10407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:27.015652 10407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:27.015678 10407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:27.017143 10407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:39993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:27.017343 10407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:27.017588 10407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:08:27.022806 10407 server_base.cc:1034] running on GCE node
W20260926 04:08:27.022790 10415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.022766 10413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.022771 10412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:27.023186 10407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:27.023432 10407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:27.024607 10407 hybrid_clock.cc:648] HybridClock initialized: now 1790395707024592 us; error 31 us; skew 500 ppm
I20260926 04:08:27.026013 10407 webserver.cc:460] Webserver started at http://127.0.213.195:34931/ using document root <none> and password file <none>
I20260926 04:08:27.026262 10407 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:27.026340 10407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:27.026499 10407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:27.027849 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "1155d8cb577c4f4887c407ef877bdc26"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "ee76159687c4dd380794b4724f18b770"
server_key_iv: "0e57e6cdf81fe7bfd6c8775090ccc1c8"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.028246 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "1155d8cb577c4f4887c407ef877bdc26"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "ee76159687c4dd380794b4724f18b770"
server_key_iv: "0e57e6cdf81fe7bfd6c8775090ccc1c8"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.029649 10407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:27.030457 10421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.030630 10407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:27.030735 10407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "1155d8cb577c4f4887c407ef877bdc26"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "ee76159687c4dd380794b4724f18b770"
server_key_iv: "0e57e6cdf81fe7bfd6c8775090ccc1c8"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.030826 10407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:27.050449 10407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:27.050793 10407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:27.050951 10407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:27.051201 10407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:27.051553 10407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:27.051608 10407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.051643 10407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:27.051690 10407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.057611 10407 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:44515
I20260926 04:08:27.057653 10534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:44515 every 8 connection(s)
I20260926 04:08:27.057993 10407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:08:27.063752 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.058748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:38091 (local address 127.0.213.254:39993)
0926 04:08:27.058816 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.058818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.059050 (+ 232us) server_negotiation.cc:407] Connection header received
0926 04:08:27.059659 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.059660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.059672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.059740 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.060541 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:27.061041 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:27.062106 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:27.062322 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:27.062403 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:27.063441 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:27.063442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:27.063443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:27.063448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:27.063464 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:27.063468 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:27.063541 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:27.063639 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 04:08:27.063668 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:08:27.063963 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.058714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:39993 (local address 127.0.213.195:38091)
0926 04:08:27.058930 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:27.058965 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:27.059551 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:27.059820 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:27.059824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:27.059835 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:27.060427 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:27.060432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:27.061142 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:27.061143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:27.062007 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:27.062009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:27.062409 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:27.062410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:27.062576 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:27.063245 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:08:27.063246 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:27.063276 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:08:27.063322 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:27.063541 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:27.063547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:27.063551 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:08:27.063627 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 04:08:27.063734 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260926 04:08:27.064502 10535 heartbeater.cc:347] Connected to a master server at 127.0.213.254:39993
I20260926 04:08:27.064611 10535 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:27.064805 10535 heartbeater.cc:511] Master 127.0.213.254:39993 requested a full tablet report, sending...
I20260926 04:08:27.065377 10085 ts_manager.cc:194] Registered new tserver with Master: 1155d8cb577c4f4887c407ef877bdc26 (127.0.213.195:44515)
I20260926 04:08:27.065783 10085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.195:38091
I20260926 04:08:27.067410 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10407
I20260926 04:08:27.067524 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:08:27.067732 855 external_mini_cluster.cc:1428] Setting key c45c3fbcadeef7122dbe9e5865329d5a
I20260926 04:08:27.068608 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 04:08:27.070405 10125 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:27.069938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.0.213.254:39993)
0926 04:08:27.069986 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.069988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.070143 (+ 155us) server_negotiation.cc:407] Connection header received
0926 04:08:27.070169 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.070170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.070180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.070241 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.070350 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46466: BlockingRecv error: recv got EOF from 127.0.0.1:46466 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:08:27.070485 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10145
I20260926 04:08:27.075328 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10276
I20260926 04:08:27.079834 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10407
I20260926 04:08:27.084128 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10054
2026-09-26T04:08:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (602 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T04:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:27Z Disabled control of system clock
I20260926 04:08:27.111043 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36593
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:36593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 04:08:27.190977 10546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:27.191193 10546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:27.191215 10546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:27.192337 10546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:27.192374 10546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:27.192387 10546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:27.192400 10546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:27.192412 10546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:27.193712 10546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:36593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:27.193913 10546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:27.194149 10546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:27.198951 10554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.198951 10552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.198964 10551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:27.199201 10546 server_base.cc:1034] running on GCE node
I20260926 04:08:27.199457 10546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:27.199698 10546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:27.200891 10546 hybrid_clock.cc:648] HybridClock initialized: now 1790395707200878 us; error 30 us; skew 500 ppm
I20260926 04:08:27.202157 10546 webserver.cc:460] Webserver started at http://127.0.213.254:43581/ using document root <none> and password file <none>
I20260926 04:08:27.202419 10546 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:27.202494 10546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:27.202654 10546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:27.203784 10546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "73fa1e1720544aa7a3d103fb96badcc6"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "5e39d846bd5c4df065535f9b1c3c072b"
server_key_iv: "6c8c936875f99eb15803a6f2f01ee3a6"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.204219 10546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "73fa1e1720544aa7a3d103fb96badcc6"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "5e39d846bd5c4df065535f9b1c3c072b"
server_key_iv: "6c8c936875f99eb15803a6f2f01ee3a6"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.205571 10546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:27.206357 10560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.206537 10546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:08:27.206640 10546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "73fa1e1720544aa7a3d103fb96badcc6"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "5e39d846bd5c4df065535f9b1c3c072b"
server_key_iv: "6c8c936875f99eb15803a6f2f01ee3a6"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.206732 10546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:27.210318 10546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:27.210593 10546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:27.210748 10546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:27.214751 10546 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:36593
I20260926 04:08:27.214797 10612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:36593 every 8 connection(s)
I20260926 04:08:27.215152 10546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:27.215325 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10546
I20260926 04:08:27.215409 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:08:27.215631 855 external_mini_cluster.cc:1428] Setting key 7413f26c977667da4f7975b136162d01
I20260926 04:08:27.215945 10613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:27.218429 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.216197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.213.254:36593)
0926 04:08:27.216447 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.216458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.216500 (+ 42us) server_negotiation.cc:407] Connection header received
0926 04:08:27.217105 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.217115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.217169 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.217325 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.217994 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:27.217998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:27.217998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:27.218003 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:27.218049 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:27.218062 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:27.218138 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:27.218191 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 04:08:27.218247 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260926 04:08:27.219537 10613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6: Bootstrap starting.
I20260926 04:08:27.220050 10613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:27.220348 10613 log.cc:824] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:27.220947 10613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6: No bootstrap required, opened a new log
I20260926 04:08:27.222214 10613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fa1e1720544aa7a3d103fb96badcc6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36593 } }
I20260926 04:08:27.222352 10613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:27.222375 10613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fa1e1720544aa7a3d103fb96badcc6, State: Initialized, Role: FOLLOWER
I20260926 04:08:27.222535 10613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fa1e1720544aa7a3d103fb96badcc6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36593 } }
I20260926 04:08:27.222643 10613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:27.222671 10613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:27.222707 10613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:27.223173 10613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fa1e1720544aa7a3d103fb96badcc6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36593 } }
I20260926 04:08:27.223263 10613 leader_election.cc:302] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73fa1e1720544aa7a3d103fb96badcc6; no voters:
I20260926 04:08:27.223384 10613 leader_election.cc:288] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:08:27.223636 10613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:27.223641 10618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:27.223806 10618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 73fa1e1720544aa7a3d103fb96badcc6, State: Running, Role: LEADER
I20260926 04:08:27.223919 10618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fa1e1720544aa7a3d103fb96badcc6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36593 } }
I20260926 04:08:27.224351 10619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73fa1e1720544aa7a3d103fb96badcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fa1e1720544aa7a3d103fb96badcc6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36593 } } }
I20260926 04:08:27.224449 10619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:27.224334 10620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fa1e1720544aa7a3d103fb96badcc6. Latest consensus state: current_term: 1 leader_uuid: "73fa1e1720544aa7a3d103fb96badcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fa1e1720544aa7a3d103fb96badcc6" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36593 } } }
I20260926 04:08:27.224838 10620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:27.224855 10627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:27.225467 10627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:27.226943 10627 catalog_manager.cc:1347] Generated new cluster ID: 6e4e3498e774489aa153a3c80081c901
I20260926 04:08:27.226997 10627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:08:27.257870 10627 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:08:27.258096 10627 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:27.269032 10627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73fa1e1720544aa7a3d103fb96badcc6: Generated new TSK 0
I20260926 04:08:27.269277 10627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:08:27.273568 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36593
--builtin_ntp_servers=127.0.213.212:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 04:08:27.366067 10637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:27.366294 10637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:27.366318 10637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:27.367550 10637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:27.367597 10637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:27.367624 10637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:27.369062 10637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:27.369333 10637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:27.369692 10637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:27.375907 10643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.375906 10642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.376150 10645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:27.376256 10637 server_base.cc:1034] running on GCE node
I20260926 04:08:27.376479 10637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:27.376767 10637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:27.377974 10637 hybrid_clock.cc:648] HybridClock initialized: now 1790395707377941 us; error 49 us; skew 500 ppm
I20260926 04:08:27.379505 10637 webserver.cc:460] Webserver started at http://127.0.213.193:38575/ using document root <none> and password file <none>
I20260926 04:08:27.379789 10637 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:27.379863 10637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:27.380009 10637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:27.381232 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "0ea7b6c995c24353be87f6099c411aa4"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "4f24638a39e330e53a785c5ba14dfdb9"
server_key_iv: "794d320ffbc4ddd742d1c922bcbd14de"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.381600 10637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea7b6c995c24353be87f6099c411aa4"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "4f24638a39e330e53a785c5ba14dfdb9"
server_key_iv: "794d320ffbc4ddd742d1c922bcbd14de"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.383009 10637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:08:27.383941 10651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.384183 10637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:27.384282 10637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "0ea7b6c995c24353be87f6099c411aa4"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "4f24638a39e330e53a785c5ba14dfdb9"
server_key_iv: "794d320ffbc4ddd742d1c922bcbd14de"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.384374 10637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:27.387850 10637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:27.388216 10637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:27.388370 10637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:27.388592 10637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:27.388907 10637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:27.388962 10637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.389002 10637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:27.389030 10637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.395262 10637 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:42815
I20260926 04:08:27.395347 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:42815 every 8 connection(s)
I20260926 04:08:27.395732 10637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:27.398788 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.396488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:34671 (local address 127.0.213.254:36593)
0926 04:08:27.396553 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.396555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.396812 (+ 257us) server_negotiation.cc:407] Connection header received
0926 04:08:27.397295 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.397296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.397303 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.397356 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.398464 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:27.398466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:27.398466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:27.398470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:27.398485 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:27.398489 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:27.398547 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:27.398718 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 04:08:27.398733 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:08:27.398996 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.396489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36593 (local address 127.0.213.193:34671)
0926 04:08:27.396685 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:27.396721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:27.397201 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:27.397407 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:27.397410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:27.397418 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:27.398285 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:08:27.398286 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:27.398322 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:08:27.398351 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:27.398597 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:27.398603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:27.398605 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:08:27.398695 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 04:08:27.398756 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260926 04:08:27.399407 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10637
I20260926 04:08:27.399510 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:08:27.399626 10765 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36593
I20260926 04:08:27.399742 855 external_mini_cluster.cc:1428] Setting key 650e49a013c91acf105276718b67d793
I20260926 04:08:27.399741 10765 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:27.399919 10765 heartbeater.cc:511] Master 127.0.213.254:36593 requested a full tablet report, sending...
I20260926 04:08:27.400496 10576 ts_manager.cc:194] Registered new tserver with Master: 0ea7b6c995c24353be87f6099c411aa4 (127.0.213.193:42815)
I20260926 04:08:27.400831 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36593
--builtin_ntp_servers=127.0.213.212:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 04:08:27.482976 10768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:27.483218 10768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:27.483258 10768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:27.484504 10768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:27.484565 10768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:27.484606 10768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:27.485965 10768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:27.486227 10768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:27.486507 10768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:27.491578 10776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.491636 10773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:27.491784 10768 server_base.cc:1034] running on GCE node
W20260926 04:08:27.491626 10774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:27.492148 10768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:27.492427 10768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:27.493573 10768 hybrid_clock.cc:648] HybridClock initialized: now 1790395707493562 us; error 35 us; skew 500 ppm
I20260926 04:08:27.494971 10768 webserver.cc:460] Webserver started at http://127.0.213.194:39297/ using document root <none> and password file <none>
I20260926 04:08:27.495234 10768 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:27.495282 10768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:27.495452 10768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:27.496604 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "8b24da17fdef493dbdf0cb833c4a63d7"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "8001d7c40affbe8e62c42b6770edbb7e"
server_key_iv: "0fef572577e3cb7463a8b7c15792afc6"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.496954 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "8b24da17fdef493dbdf0cb833c4a63d7"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "8001d7c40affbe8e62c42b6770edbb7e"
server_key_iv: "0fef572577e3cb7463a8b7c15792afc6"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.498340 10768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:08:27.499377 10782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.499574 10768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:27.499692 10768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "8b24da17fdef493dbdf0cb833c4a63d7"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "8001d7c40affbe8e62c42b6770edbb7e"
server_key_iv: "0fef572577e3cb7463a8b7c15792afc6"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.499780 10768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:27.503214 10768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:27.503479 10768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:27.503631 10768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:27.503857 10768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:27.504225 10768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:27.504276 10768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.504314 10768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:27.504343 10768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.510284 10768 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:41207
I20260926 04:08:27.510329 10895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:41207 every 8 connection(s)
I20260926 04:08:27.510684 10768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:08:27.513765 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.511517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:35705 (local address 127.0.213.254:36593)
0926 04:08:27.511601 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.511603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.511774 (+ 171us) server_negotiation.cc:407] Connection header received
0926 04:08:27.512340 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.512341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.512346 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.512410 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.513565 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:27.513566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:27.513566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:27.513571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:27.513589 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:27.513594 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:27.513652 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:27.513701 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 04:08:27.513713 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 04:08:27.513970 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.511469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36593 (local address 127.0.213.194:35705)
0926 04:08:27.511652 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:27.511686 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:27.512231 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:27.512467 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:27.512470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:27.512478 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:27.513405 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:08:27.513405 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:27.513434 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:08:27.513462 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:27.513650 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:27.513653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:27.513655 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:08:27.513708 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:08:27.513761 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260926 04:08:27.514478 10896 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36593
I20260926 04:08:27.514560 10896 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:27.514724 10896 heartbeater.cc:511] Master 127.0.213.254:36593 requested a full tablet report, sending...
I20260926 04:08:27.515089 10576 ts_manager.cc:194] Registered new tserver with Master: 8b24da17fdef493dbdf0cb833c4a63d7 (127.0.213.194:41207)
I20260926 04:08:27.515799 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10768
I20260926 04:08:27.515893 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:08:27.516178 855 external_mini_cluster.cc:1428] Setting key aa2bfdee20d594a448ee014d5ac79154
I20260926 04:08:27.517226 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36593
--builtin_ntp_servers=127.0.213.212:45357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 04:08:27.597627 10899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:27.597849 10899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:27.597872 10899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:27.599092 10899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:27.599133 10899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:27.599160 10899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:27.600605 10899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:27.600865 10899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:27.601157 10899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:27.606482 10904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.606477 10905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:27.606482 10907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:27.606775 10899 server_base.cc:1034] running on GCE node
I20260926 04:08:27.607028 10899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:27.607295 10899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:27.608486 10899 hybrid_clock.cc:648] HybridClock initialized: now 1790395707608482 us; error 63 us; skew 500 ppm
I20260926 04:08:27.609890 10899 webserver.cc:460] Webserver started at http://127.0.213.195:36931/ using document root <none> and password file <none>
I20260926 04:08:27.610180 10899 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:27.610280 10899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:27.610440 10899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:27.611564 10899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "06e057b7b2a446f0870532566a0eb1e8"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "377489d3aa918063e56338855b74e697"
server_key_iv: "335c165e8d5ada431bee1d4f65f62a28"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.611946 10899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "06e057b7b2a446f0870532566a0eb1e8"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "377489d3aa918063e56338855b74e697"
server_key_iv: "335c165e8d5ada431bee1d4f65f62a28"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.613394 10899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:27.614321 10913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.614573 10899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:27.614686 10899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "06e057b7b2a446f0870532566a0eb1e8"
format_stamp: "Formatted at 2026-09-26 04:08:27 on dist-test-slave-xf24"
server_key: "377489d3aa918063e56338855b74e697"
server_key_iv: "335c165e8d5ada431bee1d4f65f62a28"
server_key_version: "encryptionkey@0"
I20260926 04:08:27.614784 10899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:27.618377 10899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:27.618670 10899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:27.618821 10899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:27.619046 10899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:27.619379 10899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:27.619436 10899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.619477 10899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:27.619504 10899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:27.625913 10899 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:33849
I20260926 04:08:27.625962 11026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:33849 every 8 connection(s)
I20260926 04:08:27.626333 10899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:08:27.629431 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.627144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:44031 (local address 127.0.213.254:36593)
0926 04:08:27.627217 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.627219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.627436 (+ 217us) server_negotiation.cc:407] Connection header received
0926 04:08:27.627879 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.627880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.627886 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.627942 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.629219 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:27.629221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:27.629221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:08:27.629225 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:27.629238 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:08:27.629242 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:27.629305 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:27.629354 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 04:08:27.629365 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:08:27.629597 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:27.627153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36593 (local address 127.0.213.195:44031)
0926 04:08:27.627307 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:27.627340 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:27.627788 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:27.627995 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:27.627998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:27.628005 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:27.629053 (+ 1048us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:08:27.629053 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:27.629082 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:08:27.629110 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:27.629305 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:27.629308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:27.629310 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:08:27.629361 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 04:08:27.629406 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260926 04:08:27.630076 11027 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36593
I20260926 04:08:27.630169 11027 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:27.630275 11027 heartbeater.cc:511] Master 127.0.213.254:36593 requested a full tablet report, sending...
I20260926 04:08:27.630610 10576 ts_manager.cc:194] Registered new tserver with Master: 06e057b7b2a446f0870532566a0eb1e8 (127.0.213.195:33849)
I20260926 04:08:27.632000 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 10899
I20260926 04:08:27.632155 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:08:27.632330 855 external_mini_cluster.cc:1428] Setting key 1d5ea3f980bbaa49cf4912af715eccbd
I20260926 04:08:27.633167 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 04:08:27.634970 10616 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:27.634477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.0.213.254:36593)
0926 04:08:27.634548 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:27.634551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:27.634647 (+ 96us) server_negotiation.cc:407] Connection header received
0926 04:08:27.634709 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:27.634710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:27.634715 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:27.634765 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:27.634911 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50444: BlockingRecv error: recv got EOF from 127.0.0.1:50444 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 04:08:27.635221 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10637
I20260926 04:08:27.639959 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10768
I20260926 04:08:27.644665 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10899
I20260926 04:08:27.648954 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 10546
2026-09-26T04:08:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (564 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:08:27 dist-test-slave-xf24 krb5kdc[11037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:08:27 dist-test-slave-xf24 krb5kdc[11037](info): set up 2 sockets
Sep 26 04:08:27 dist-test-slave-xf24 krb5kdc[11037](info): commencing operation
krb5kdc: starting...
W20260926 04:08:28.689213 855 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:08:28 dist-test-slave-xf24 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:08:28.815542 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36281
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:28.898197 11050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:28.898440 11050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:28.898476 11050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:28.899626 11050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:08:28.899696 11050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:28.899721 11050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:28.899742 11050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:08:28.899775 11050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:08:28.901312 11050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:36281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:28.901590 11050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:28.901819 11050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:28.906677 11056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:28.906663 11055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:28.906663 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:28.907094 11050 server_base.cc:1034] running on GCE node
I20260926 04:08:28.907331 11050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:28.907575 11050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:28.908746 11050 hybrid_clock.cc:648] HybridClock initialized: now 1790395708908732 us; error 32 us; skew 500 ppm
Sep 26 04:08:28 dist-test-slave-xf24 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:28.911535 11050 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:08:28.912040 11050 webserver.cc:460] Webserver started at http://127.0.213.254:38157/ using document root <none> and password file <none>
I20260926 04:08:28.912352 11050 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:28.912420 11050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:28.912570 11050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:28.913605 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "0ede6d001cfe44e6bbc0a77e35c7c608"
format_stamp: "Formatted at 2026-09-26 04:08:28 on dist-test-slave-xf24"
server_key: "a361da19115baa743e2a6fc3e6786d5e"
server_key_iv: "ca597460c4c52e7c85628d08ce536515"
server_key_version: "encryptionkey@0"
I20260926 04:08:28.913967 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "0ede6d001cfe44e6bbc0a77e35c7c608"
format_stamp: "Formatted at 2026-09-26 04:08:28 on dist-test-slave-xf24"
server_key: "a361da19115baa743e2a6fc3e6786d5e"
server_key_iv: "ca597460c4c52e7c85628d08ce536515"
server_key_version: "encryptionkey@0"
I20260926 04:08:28.915285 11050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:08:28.916051 11065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:28.916283 11050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:08:28.916383 11050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "0ede6d001cfe44e6bbc0a77e35c7c608"
format_stamp: "Formatted at 2026-09-26 04:08:28 on dist-test-slave-xf24"
server_key: "a361da19115baa743e2a6fc3e6786d5e"
server_key_iv: "ca597460c4c52e7c85628d08ce536515"
server_key_version: "encryptionkey@0"
I20260926 04:08:28.916468 11050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:28.939779 11050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:28.941982 11050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:28.942152 11050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:28.946106 11050 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:36281
I20260926 04:08:28.946143 11117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:36281 every 8 connection(s)
I20260926 04:08:28.946514 11050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:08:28.947526 11118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:28.949875 11118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608: Bootstrap starting.
I20260926 04:08:28.950470 11118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:28.950762 11118 log.cc:824] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:28.950958 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 11050
I20260926 04:08:28.951113 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:08:28.951349 855 external_mini_cluster.cc:1428] Setting key 894bf0333b71805e140045e9cc524774
I20260926 04:08:28.951594 11118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608: No bootstrap required, opened a new log
I20260926 04:08:28.953734 11118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36281 } }
I20260926 04:08:28.953928 11118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:28.953974 11118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ede6d001cfe44e6bbc0a77e35c7c608, State: Initialized, Role: FOLLOWER
I20260926 04:08:28.954110 11118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36281 } }
I20260926 04:08:28.954206 11118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:08:28.954242 11118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:08:28.954289 11118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:28.955014 11118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36281 } }
I20260926 04:08:28.955122 11118 leader_election.cc:302] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ede6d001cfe44e6bbc0a77e35c7c608; no voters:
I20260926 04:08:28.955278 11118 leader_election.cc:288] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:08:28.955344 11123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:28.955514 11118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:08:28.955530 11123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [term 1 LEADER]: Becoming Leader. State: Replica: 0ede6d001cfe44e6bbc0a77e35c7c608, State: Running, Role: LEADER
I20260926 04:08:28.955698 11123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36281 } }
I20260926 04:08:28.956102 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36281 } } }
I20260926 04:08:28.956188 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:28.956343 11125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ede6d001cfe44e6bbc0a77e35c7c608. Latest consensus state: current_term: 1 leader_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ede6d001cfe44e6bbc0a77e35c7c608" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36281 } } }
I20260926 04:08:28.956394 11125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608 [sys.catalog]: This master's current role is: LEADER
I20260926 04:08:28.956664 11132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:08:28.957203 11132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:08:28.959154 11132 catalog_manager.cc:1347] Generated new cluster ID: 8c51157799064fc3a9062473efa9451d
I20260926 04:08:28.959216 11132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:08:28 dist-test-slave-xf24 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:28.963263 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:28.951893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.0.213.254:36281)
0926 04:08:28.952123 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:28.952133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:28.952167 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:08:28.952976 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:28.952994 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:28.953060 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:28.953216 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:28.953845 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:28.955535 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:28.956777 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:28.957436 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:28.957559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:28.960104 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:28.960108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:28.960108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:28.960120 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:28.961774 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:28.962143 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:28.962146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:28.962146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:28.962231 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:28.962443 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:28.962444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:28.962444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:28.962677 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:28.962873 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:28.962964 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 04:08:28.963118 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260926 04:08:28.972182 11132 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:08:28.972744 11132 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:08:28.978235 11132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608: Generated new TSK 0
I20260926 04:08:28.978417 11132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:08:29.012228 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36281
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:29.115005 11144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:29.115240 11144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:29.115266 11144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:29.116560 11144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:29.116611 11144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:29.116641 11144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:08:29.118352 11144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:29.118637 11144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:29.118912 11144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:29.124395 11152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:29.124409 11150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:29.124406 11149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:29.124683 11144 server_base.cc:1034] running on GCE node
I20260926 04:08:29.124886 11144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:29.125170 11144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:29.126324 11144 hybrid_clock.cc:648] HybridClock initialized: now 1790395709126301 us; error 47 us; skew 500 ppm
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:29.129161 11144 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:08:29.129628 11144 webserver.cc:460] Webserver started at http://127.0.213.193:36771/ using document root <none> and password file <none>
I20260926 04:08:29.129812 11144 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:29.129851 11144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:29.129938 11144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:29.130950 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "e603db515647413e98937bbae296b8e5"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "c535863013c2a3befd3311cc681ad6f2"
server_key_iv: "71ecbe4d07989bc034608c43da682dbe"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.131255 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "e603db515647413e98937bbae296b8e5"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "c535863013c2a3befd3311cc681ad6f2"
server_key_iv: "71ecbe4d07989bc034608c43da682dbe"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.132649 11144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:29.133385 11159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.133615 11144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:08:29.133718 11144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "e603db515647413e98937bbae296b8e5"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "c535863013c2a3befd3311cc681ad6f2"
server_key_iv: "71ecbe4d07989bc034608c43da682dbe"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.133800 11144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:29.162994 11144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:29.165315 11144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:29.165511 11144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:29.165786 11144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:29.166119 11144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:29.166177 11144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.166218 11144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:29.166266 11144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.172369 11144 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:44071
I20260926 04:08:29.172432 11272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:44071 every 8 connection(s)
I20260926 04:08:29.172775 11144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:08:29.178226 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 11144
I20260926 04:08:29.178318 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:08:29.178542 855 external_mini_cluster.cc:1428] Setting key ef1fac1a39e88994d7193be64230fcd8
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:29.182808 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.173584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:49801 (local address 127.0.213.254:36281)
0926 04:08:29.173658 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.173660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.173840 (+ 180us) server_negotiation.cc:407] Connection header received
0926 04:08:29.174290 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.174292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.174302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.174368 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:29.175365 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.175884 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.177020 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.177213 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.177274 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.179622 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:29.179624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:29.179624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:29.179632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:29.181305 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.181830 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.181830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.181831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.181879 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.182153 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.182153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.182154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.182288 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:29.182340 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.182674 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.182741 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:08:29.183113 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.173491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36281 (local address 127.0.213.193:49801)
0926 04:08:29.173682 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.173724 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.174196 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.174443 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.174447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.175010 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:29.175245 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.175250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.176021 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.176021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.176904 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.176906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.177291 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.177292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.177392 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.177904 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:29.177913 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:29.179514 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:29.181391 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:29.181400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:29.181402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:29.181753 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:29.181943 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:29.181944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:29.181944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:29.182090 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:29.182352 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:29.182358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:29.182527 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.182654 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.182840 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260926 04:08:29.183748 11273 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36281
I20260926 04:08:29.183851 11273 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:29.184113 11273 heartbeater.cc:511] Master 127.0.213.254:36281 requested a full tablet report, sending...
I20260926 04:08:29.184676 11082 ts_manager.cc:194] Registered new tserver with Master: e603db515647413e98937bbae296b8e5 (127.0.213.193:44071)
I20260926 04:08:29.185321 11082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:49801
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:08:29.206526 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36281
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:29.287091 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:29.287343 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:29.287393 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:29.288638 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:29.288702 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:29.288750 11278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:08:29.290196 11278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:29.290467 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:29.290701 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:29.295794 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:29.295822 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:29.295992 11278 server_base.cc:1034] running on GCE node
W20260926 04:08:29.295825 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:29.296295 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:29.296542 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:29.297700 11278 hybrid_clock.cc:648] HybridClock initialized: now 1790395709297696 us; error 36 us; skew 500 ppm
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:29.300330 11278 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:08:29.300741 11278 webserver.cc:460] Webserver started at http://127.0.213.194:35371/ using document root <none> and password file <none>
I20260926 04:08:29.300944 11278 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:29.300987 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:29.301082 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:29.302115 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "b1c107e7dfbf4a8590cb954b18e7741a"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "c16e3577d1eef218310b4b1e9185a405"
server_key_iv: "8dcacdd0c6df80521f3dc9a609d68fd3"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.302430 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "b1c107e7dfbf4a8590cb954b18e7741a"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "c16e3577d1eef218310b4b1e9185a405"
server_key_iv: "8dcacdd0c6df80521f3dc9a609d68fd3"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.303762 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:08:29.304620 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.304814 11278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:08:29.304899 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "b1c107e7dfbf4a8590cb954b18e7741a"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "c16e3577d1eef218310b4b1e9185a405"
server_key_iv: "8dcacdd0c6df80521f3dc9a609d68fd3"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.304955 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:29.328344 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:29.330786 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:29.330909 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:29.331102 11278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:29.331486 11278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:29.331527 11278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.331552 11278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:29.331565 11278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.337525 11278 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38523
I20260926 04:08:29.337653 11406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38523 every 8 connection(s)
I20260926 04:08:29.337949 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:08:29.341491 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 11278
I20260926 04:08:29.341578 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:08:29.341809 855 external_mini_cluster.cc:1428] Setting key eb441f5dfbc4d8321b216134bbaf8e2f
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:29.348508 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.338742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:53069 (local address 127.0.213.254:36281)
0926 04:08:29.338802 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.338804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.339038 (+ 234us) server_negotiation.cc:407] Connection header received
0926 04:08:29.339556 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.339557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.339568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.339656 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:29.340650 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.341150 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.342366 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.342636 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.342695 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.345446 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:29.345448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:29.345449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:29.345457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:29.347009 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.347458 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.347459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.347459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.347524 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.347787 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.347788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.347788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.347911 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:29.347981 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.348330 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.348414 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 04:08:29.348728 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.338716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36281 (local address 127.0.213.194:53069)
0926 04:08:29.338905 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.338950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.339443 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.339749 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.339753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.340365 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:29.340559 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.340563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.341240 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.341241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.342281 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.342283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.342727 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.342728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.342879 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.343414 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:29.343424 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:29.345342 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:29.347116 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:29.347125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:29.347127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:29.347367 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:29.347590 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:29.347590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:29.347591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:29.347714 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:29.347982 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:29.347989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:29.348187 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.348299 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.348465 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260926 04:08:29.349202 11407 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36281
I20260926 04:08:29.349330 11407 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:29.349520 11407 heartbeater.cc:511] Master 127.0.213.254:36281 requested a full tablet report, sending...
I20260926 04:08:29.350045 11082 ts_manager.cc:194] Registered new tserver with Master: b1c107e7dfbf4a8590cb954b18e7741a (127.0.213.194:38523)
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:08:29.350461 11082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:53069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:08:29.372454 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36281
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:08:29.454591 11412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:08:29.454828 11412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:08:29.454865 11412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:08:29.456317 11412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:08:29.456382 11412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:08:29.456429 11412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:08:29.458101 11412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:08:29.458379 11412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:08:29.458623 11412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:08:29.463896 11420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:29.463908 11417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:08:29.463912 11418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:08:29.464174 11412 server_base.cc:1034] running on GCE node
I20260926 04:08:29.464380 11412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:08:29.464596 11412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:08:29.465765 11412 hybrid_clock.cc:648] HybridClock initialized: now 1790395709465734 us; error 42 us; skew 500 ppm
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:08:29.468523 11412 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:08:29.469009 11412 webserver.cc:460] Webserver started at http://127.0.213.195:41827/ using document root <none> and password file <none>
I20260926 04:08:29.469290 11412 fs_manager.cc:359] Metadata directory not provided
I20260926 04:08:29.469362 11412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:08:29.469525 11412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:08:29.470643 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "51c73ed39495404b9000f8ecf3bdd24c"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "60e3e2dc950b5f38efe7a6d62df5f3dd"
server_key_iv: "c6abbc9da3704e73de045951328e542a"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.471037 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "51c73ed39495404b9000f8ecf3bdd24c"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "60e3e2dc950b5f38efe7a6d62df5f3dd"
server_key_iv: "c6abbc9da3704e73de045951328e542a"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.472501 11412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:08:29.473296 11427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.473500 11412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:08:29.473616 11412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "51c73ed39495404b9000f8ecf3bdd24c"
format_stamp: "Formatted at 2026-09-26 04:08:29 on dist-test-slave-xf24"
server_key: "60e3e2dc950b5f38efe7a6d62df5f3dd"
server_key_iv: "c6abbc9da3704e73de045951328e542a"
server_key_version: "encryptionkey@0"
I20260926 04:08:29.473693 11412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:08:29.504367 11412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:08:29.506688 11412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:08:29.506881 11412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:08:29.507134 11412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:08:29.507503 11412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:08:29.507581 11412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.507616 11412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:08:29.507638 11412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:08:29.513635 11412 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:37813
I20260926 04:08:29.513717 11540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:37813 every 8 connection(s)
I20260926 04:08:29.514050 11412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:08:29.517666 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 11412
I20260926 04:08:29.517774 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:08:29.517969 855 external_mini_cluster.cc:1428] Setting key 4ac9c8f6bf217512c5cd8cfc07dfd9f7
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:08:29.524379 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.514854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:42729 (local address 127.0.213.254:36281)
0926 04:08:29.514947 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.514950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.515185 (+ 235us) server_negotiation.cc:407] Connection header received
0926 04:08:29.515701 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.515702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.515712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.515775 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:29.516954 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.517507 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.519080 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.519337 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.519416 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.521824 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:29.521826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:29.521826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:29.521832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:29.523097 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.523465 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.523466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.523466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.523514 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.523723 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.523724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.523724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.523823 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:29.523873 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.524253 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.524314 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:08:29.524533 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.514856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36281 (local address 127.0.213.195:42729)
0926 04:08:29.515037 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.515083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.515613 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.515858 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.515861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.516640 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:08:29.516854 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.516858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.517609 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.517611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.519014 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.519016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.519421 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.519423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.519563 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.520206 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:08:29.520216 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:08:29.521736 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:08:29.523171 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:29.523177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:29.523179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:29.523398 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:29.523571 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:08:29.523572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:08:29.523572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:08:29.523668 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:08:29.523883 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:08:29.523888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:08:29.524044 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.524197 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.524317 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260926 04:08:29.525142 11541 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36281
I20260926 04:08:29.525229 11541 heartbeater.cc:464] Registering TS with master...
I20260926 04:08:29.525367 11541 heartbeater.cc:511] Master 127.0.213.254:36281 requested a full tablet report, sending...
I20260926 04:08:29.525727 11082 ts_manager.cc:194] Registered new tserver with Master: 51c73ed39495404b9000f8ecf3bdd24c (127.0.213.195:37813)
I20260926 04:08:29.526098 11082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:42729
I20260926 04:08:29.529502 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:08:29.537147 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.530796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.0.213.254:36281)
0926 04:08:29.530859 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.530861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.530994 (+ 133us) server_negotiation.cc:407] Connection header received
0926 04:08:29.531018 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.531019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.531028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.531086 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:08:29.531655 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.532251 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.533067 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.533254 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.533318 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.534526 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:08:29.534528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:08:29.534528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:08:29.534535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:08:29.535849 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.536323 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.536325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.536325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.536382 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:08:29.536544 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:08:29.536545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:08:29.536545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:08:29.536693 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:08:29.536768 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.536883 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.536967 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 04:08:29.538879 11082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:08:29.540688 11082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:08:29.547757 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.543479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.0.213.195:37813)
0926 04:08:29.543632 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.543637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.543775 (+ 138us) server_negotiation.cc:407] Connection header received
0926 04:08:29.544130 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.544133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.544150 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.544224 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:29.544421 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.545471 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.546567 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.547303 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.547384 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.547446 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.547627 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.547669 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260926 04:08:29.547755 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.543388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:37813 (local address 127.0.0.1:46578)
0926 04:08:29.543665 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.543710 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.543789 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.544309 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.544310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.544311 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:29.544347 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.544348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.545555 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.545556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.546483 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.546485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.547407 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.547408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.547501 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.547518 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.547601 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.547664 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260926 04:08:29.548159 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.543477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.213.193:44071)
0926 04:08:29.543730 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.543734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.543799 (+ 65us) server_negotiation.cc:407] Connection header received
0926 04:08:29.543878 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.543881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.543896 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.543973 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:29.544241 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.545394 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.546768 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.547433 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.547503 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.547540 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.548011 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.548081 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260926 04:08:29.547753 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.543393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44071 (local address 127.0.0.1:50860)
0926 04:08:29.543674 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.543712 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.543789 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.543982 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.543986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.543988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:29.544136 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.544139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.545501 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.545502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.546687 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.546688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.547533 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.547534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.547597 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.547600 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.547651 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.547686 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260926 04:08:29.548784 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.543622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38523 (local address 127.0.0.1:49030)
0926 04:08:29.543997 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.544008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.544093 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.544399 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.544400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.544402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:29.544450 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.544452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.545336 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.545337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.546679 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.546682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.548544 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.548545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.548605 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.548610 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.548676 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.548720 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":50,"thread_start_us":69,"threads_started":1}
I20260926 04:08:29.548671 11475 tablet_service.cc:1467] Processing CreateTablet for tablet 1955a86405044d978983380111e3f197 (DEFAULT_TABLE table=test-table [id=fe3051b1ec384219a1988bb0e907a9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:29.548849 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.543838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.0.213.194:38523)
0926 04:08:29.544016 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.544023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.544074 (+ 51us) server_negotiation.cc:407] Connection header received
0926 04:08:29.544224 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.544227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.544241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.544315 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:29.544530 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.545215 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.547594 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.548392 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.548460 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.548508 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.548693 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.548743 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260926 04:08:29.549047 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1955a86405044d978983380111e3f197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:29.549371 11207 tablet_service.cc:1467] Processing CreateTablet for tablet 1955a86405044d978983380111e3f197 (DEFAULT_TABLE table=test-table [id=fe3051b1ec384219a1988bb0e907a9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:29.549554 11341 tablet_service.cc:1467] Processing CreateTablet for tablet 1955a86405044d978983380111e3f197 (DEFAULT_TABLE table=test-table [id=fe3051b1ec384219a1988bb0e907a9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:08:29.549705 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1955a86405044d978983380111e3f197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:29.549898 11341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1955a86405044d978983380111e3f197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:08:29.552451 11558 tablet_bootstrap.cc:492] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Bootstrap starting.
I20260926 04:08:29.552932 11560 tablet_bootstrap.cc:492] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: Bootstrap starting.
I20260926 04:08:29.553041 11559 tablet_bootstrap.cc:492] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Bootstrap starting.
I20260926 04:08:29.553251 11558 tablet_bootstrap.cc:654] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:29.553622 11560 tablet_bootstrap.cc:654] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:29.553632 11559 tablet_bootstrap.cc:654] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Neither blocks nor log segments found. Creating new log.
I20260926 04:08:29.553798 11558 log.cc:824] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:29.554215 11559 log.cc:824] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:29.554215 11560 log.cc:824] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: Log is configured to *not* fsync() on all Append() calls
I20260926 04:08:29.554584 11558 tablet_bootstrap.cc:492] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: No bootstrap required, opened a new log
I20260926 04:08:29.554813 11558 ts_tablet_manager.cc:1397] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:08:29.554968 11559 tablet_bootstrap.cc:492] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: No bootstrap required, opened a new log
I20260926 04:08:29.555038 11559 ts_tablet_manager.cc:1397] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:08:29.555274 11560 tablet_bootstrap.cc:492] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: No bootstrap required, opened a new log
I20260926 04:08:29.555339 11560 ts_tablet_manager.cc:1397] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:08:29.556468 11558 raft_consensus.cc:348] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.556638 11558 raft_consensus.cc:374] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:29.556680 11558 raft_consensus.cc:729] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c73ed39495404b9000f8ecf3bdd24c, State: Initialized, Role: FOLLOWER
I20260926 04:08:29.556792 11558 consensus_queue.cc:260] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.557082 11558 ts_tablet_manager.cc:1428] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:08:29.557248 11541 heartbeater.cc:503] Master 127.0.213.254:36281 was elected leader, sending a full tablet report...
I20260926 04:08:29.557240 11560 raft_consensus.cc:348] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.557205 11559 raft_consensus.cc:348] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.557382 11560 raft_consensus.cc:374] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:29.557382 11559 raft_consensus.cc:374] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:08:29.557425 11559 raft_consensus.cc:729] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c107e7dfbf4a8590cb954b18e7741a, State: Initialized, Role: FOLLOWER
I20260926 04:08:29.557425 11560 raft_consensus.cc:729] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e603db515647413e98937bbae296b8e5, State: Initialized, Role: FOLLOWER
I20260926 04:08:29.557538 11560 consensus_queue.cc:260] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.557538 11559 consensus_queue.cc:260] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.557803 11559 ts_tablet_manager.cc:1428] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:08:29.557803 11560 ts_tablet_manager.cc:1428] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:08:29.557915 11407 heartbeater.cc:503] Master 127.0.213.254:36281 was elected leader, sending a full tablet report...
I20260926 04:08:29.557958 11273 heartbeater.cc:503] Master 127.0.213.254:36281 was elected leader, sending a full tablet report...
W20260926 04:08:29.588624 11408 tablet.cc:2369] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:08:29.673534 11274 tablet.cc:2369] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:08:29.674224 11565 raft_consensus.cc:482] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:08:29.674333 11565 raft_consensus.cc:504] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.674760 11565 leader_election.cc:288] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e603db515647413e98937bbae296b8e5 (127.0.213.193:44071), 51c73ed39495404b9000f8ecf3bdd24c (127.0.213.195:37813)
I20260926 04:08:29.677909 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.675173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:58125 (local address 127.0.213.195:37813)
0926 04:08:29.675222 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.675224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.675256 (+ 32us) server_negotiation.cc:407] Connection header received
0926 04:08:29.675330 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.675331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.675340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.675400 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:29.675578 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.676023 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.677162 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.677541 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.677639 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.677647 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.677794 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.677834 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 04:08:29.677927 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.674991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44071 (local address 127.0.213.194:40593)
0926 04:08:29.675084 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.675099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.675167 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.675328 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.675329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.675331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:29.675409 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.675411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.676102 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.676103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.677027 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.677030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.677628 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.677629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.677720 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.677723 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.677798 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.677837 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":1}
I20260926 04:08:29.677930 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.675101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:40593 (local address 127.0.213.193:44071)
0926 04:08:29.675151 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.675154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.675167 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:08:29.675245 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.675246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.675256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.675312 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:08:29.675492 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.676011 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.677113 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.677542 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.677630 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.677640 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.677813 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.677846 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:08:29.677943 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.675019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:37813 (local address 127.0.213.194:58125)
0926 04:08:29.675204 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.675213 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.675269 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.675412 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:08:29.675413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:08:29.675414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:08:29.675504 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.675506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.676185 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.676186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.677047 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:08:29.677049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.677643 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.677644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:08:29.677706 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.677709 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:08:29.677752 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 04:08:29.677812 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260926 04:08:29.678265 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1955a86405044d978983380111e3f197" candidate_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c73ed39495404b9000f8ecf3bdd24c" is_pre_election: true
I20260926 04:08:29.678400 11495 raft_consensus.cc:2454] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c107e7dfbf4a8590cb954b18e7741a in term 0.
I20260926 04:08:29.678668 11297 leader_election.cc:302] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c73ed39495404b9000f8ecf3bdd24c, b1c107e7dfbf4a8590cb954b18e7741a; no voters:
I20260926 04:08:29.678813 11565 raft_consensus.cc:2781] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:08:29.678798 11227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1955a86405044d978983380111e3f197" candidate_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e603db515647413e98937bbae296b8e5" is_pre_election: true
I20260926 04:08:29.678902 11565 raft_consensus.cc:482] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:08:29.678932 11565 raft_consensus.cc:3037] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:29.679026 11227 raft_consensus.cc:2454] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c107e7dfbf4a8590cb954b18e7741a in term 0.
I20260926 04:08:29.679719 11565 raft_consensus.cc:504] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.679939 11565 leader_election.cc:288] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [CANDIDATE]: Term 1 election: Requested vote from peers e603db515647413e98937bbae296b8e5 (127.0.213.193:44071), 51c73ed39495404b9000f8ecf3bdd24c (127.0.213.195:37813)
I20260926 04:08:29.680919 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1955a86405044d978983380111e3f197" candidate_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c73ed39495404b9000f8ecf3bdd24c"
I20260926 04:08:29.680996 11495 raft_consensus.cc:3037] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:29.681995 11495 raft_consensus.cc:2454] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c107e7dfbf4a8590cb954b18e7741a in term 1.
I20260926 04:08:29.682327 11227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1955a86405044d978983380111e3f197" candidate_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e603db515647413e98937bbae296b8e5"
I20260926 04:08:29.682415 11227 raft_consensus.cc:3037] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:08:29.682472 11297 leader_election.cc:302] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c73ed39495404b9000f8ecf3bdd24c, b1c107e7dfbf4a8590cb954b18e7741a; no voters:
I20260926 04:08:29.682580 11565 raft_consensus.cc:2781] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:08:29.682727 11565 raft_consensus.cc:686] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 1 LEADER]: Becoming Leader. State: Replica: b1c107e7dfbf4a8590cb954b18e7741a, State: Running, Role: LEADER
I20260926 04:08:29.682829 11565 consensus_queue.cc:237] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } }
I20260926 04:08:29.683395 11227 raft_consensus.cc:2454] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c107e7dfbf4a8590cb954b18e7741a in term 1.
I20260926 04:08:29.684598 11081 catalog_manager.cc:5495] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a reported cstate change: term changed from 0 to 1, leader changed from <none> to b1c107e7dfbf4a8590cb954b18e7741a (127.0.213.194). New cstate: current_term: 1 leader_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1c107e7dfbf4a8590cb954b18e7741a" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38523 } health_report { overall_health: HEALTHY } } }
I20260926 04:08:29.693929 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:08:29.690217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.213.194:38523)
0926 04:08:29.690280 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:08:29.690283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:08:29.690293 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:08:29.690363 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:08:29.690364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:08:29.690383 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:08:29.690733 (+ 350us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:08:29.690896 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.691453 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.692555 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:08:29.692924 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:08:29.693007 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:08:29.693192 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:08:29.693489 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:08:29.693573 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:08:29.693780 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 04:08:29.693804 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:08:29.696154 11495 raft_consensus.cc:1261] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 1 FOLLOWER]: Refusing update from remote peer b1c107e7dfbf4a8590cb954b18e7741a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:29.696220 11227 raft_consensus.cc:1261] T 1955a86405044d978983380111e3f197 P e603db515647413e98937bbae296b8e5 [term 1 FOLLOWER]: Refusing update from remote peer b1c107e7dfbf4a8590cb954b18e7741a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:08:29.696535 11568 consensus_queue.cc:1035] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c73ed39495404b9000f8ecf3bdd24c" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:29.696663 11565 consensus_queue.cc:1035] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e603db515647413e98937bbae296b8e5" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:08:29.699096 11571 mvcc.cc:205] Tried to move back new op lower bound from 7333460826912493568 to 7333460826861756416. Current Snapshot: MvccSnapshot[applied={T|T < 7333460826912493568}]
I20260926 04:08:29.699671 11572 mvcc.cc:205] Tried to move back new op lower bound from 7333460826912493568 to 7333460826861756416. Current Snapshot: MvccSnapshot[applied={T|T < 7333460826912493568}]
I20260926 04:08:29.700279 11573 mvcc.cc:205] Tried to move back new op lower bound from 7333460826912493568 to 7333460826861756416. Current Snapshot: MvccSnapshot[applied={T|T < 7333460826912493568}]
I20260926 04:08:29.701632 11081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:29.701761 11081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:08:29.706561 11081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ede6d001cfe44e6bbc0a77e35c7c608: Sending DeleteTablet for 3 replicas of tablet 1955a86405044d978983380111e3f197
I20260926 04:08:29.707002 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet 1955a86405044d978983380111e3f197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:29 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:46578
I20260926 04:08:29.707002 11207 tablet_service.cc:1514] Processing DeleteTablet for tablet 1955a86405044d978983380111e3f197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:29 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:50860
I20260926 04:08:29.707067 11341 tablet_service.cc:1514] Processing DeleteTablet for tablet 1955a86405044d978983380111e3f197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:08:29 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:49030
I20260926 04:08:29.707239 11580 tablet_replica.cc:330] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: stopping tablet replica
I20260926 04:08:29.707350 11580 raft_consensus.cc:2229] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:08:29.707386 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 11144
I20260926 04:08:29.707463 11582 tablet_replica.cc:330] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: stopping tablet replica
I20260926 04:08:29.707522 11580 raft_consensus.cc:2258] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:29.707629 11582 raft_consensus.cc:2229] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:08:29.707767 11582 raft_consensus.cc:2258] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:08:29.708276 11582 ts_tablet_manager.cc:1905] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:29.709246 11580 ts_tablet_manager.cc:1905] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:08:29.710351 11582 ts_tablet_manager.cc:1918] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:29.710506 11582 log.cc:1196] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/1955a86405044d978983380111e3f197
I20260926 04:08:29.710879 11582 ts_tablet_manager.cc:1939] T 1955a86405044d978983380111e3f197 P b1c107e7dfbf4a8590cb954b18e7741a: Deleting consensus metadata
I20260926 04:08:29.711163 11580 ts_tablet_manager.cc:1918] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:08:29.711239 11580 log.cc:1196] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/1955a86405044d978983380111e3f197
I20260926 04:08:29.711508 11580 ts_tablet_manager.cc:1939] T 1955a86405044d978983380111e3f197 P 51c73ed39495404b9000f8ecf3bdd24c: Deleting consensus metadata
I20260926 04:08:29.711930 11069 catalog_manager.cc:4841] TS 51c73ed39495404b9000f8ecf3bdd24c (127.0.213.195:37813): tablet 1955a86405044d978983380111e3f197 (table test-table [id=fe3051b1ec384219a1988bb0e907a9b2]) successfully deleted
W20260926 04:08:29.714584 11066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:44071 (error 108)
I20260926 04:08:29.715080 11066 catalog_manager.cc:4841] TS b1c107e7dfbf4a8590cb954b18e7741a (127.0.213.194:38523): tablet 1955a86405044d978983380111e3f197 (table test-table [id=fe3051b1ec384219a1988bb0e907a9b2]) successfully deleted
W20260926 04:08:29.715312 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:08:29.715032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:44071 (local address 127.0.0.1:50868)
0926 04:08:29.715072 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 04:08:29.715084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:08:29.715125 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:08:29.715260 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:44071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 04:08:29.715518 11066 catalog_manager.cc:4584] TS e603db515647413e98937bbae296b8e5 (127.0.213.193:44071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1955a86405044d978983380111e3f197: Network error: Client connection negotiation failed: client connection to 127.0.213.193:44071: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:08:29.715718 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 11278
I20260926 04:08:29.722821 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 11412
I20260926 04:08:29.727432 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 11050
2026-09-26T04:08:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2083 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11586](info): set up 2 sockets
Sep 26 04:08:29 dist-test-slave-xf24 krb5kdc[11586](info): commencing operation
krb5kdc: starting...
W20260926 04:08:30.785295 855 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:08:30 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:08:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:08:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/rangeradmin_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/rangeradmin_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/rangerlookup_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/rangerlookup_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
I20260926 04:08:30.963222 855 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrg1Sr7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/postgres -l logfile start
2026-09-26 04:08:33.767 UTC [11616] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 04:08:33.767 UTC [11616] LOG: listening on IPv4 address "127.0.213.212", port 44479
2026-09-26 04:08:33.771 UTC [11616] LOG: listening on Unix socket "/tmp/.s.PGSQL.44479"
2026-09-26 04:08:33.792 UTC [11619] LOG: database system was shut down at 2026-09-26 04:08:32 UTC
2026-09-26 04:08:33.796 UTC [11616] LOG: database system is ready to accept connections
I20260926 04:08:34.753048 855 mini_postgres.cc:96] Postgres bound to 44479
2026-09-26 04:08:34.759 UTC [11627] FATAL: database "slave" does not exist
127.0.213.212:44479 - accepting connections
I20260926 04:08:34.759819 855 mini_ranger.cc:161] Starting Ranger...
I20260926 04:08:34.774112 855 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 04:08:35.332305 855 mini_ranger.cc:87] Created ranger Postgres database
I20260926 04:08:35.332372 855 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-admin
2026-09-26 04:08:35,610 [I] DB FLAVOR :POSTGRES
2026-09-26 04:08:35,610 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 04:08:35,610 [I] Checking connection..
2026-09-26 04:08:35,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 04:08:35,862 [I] Checking connection passed.
2026-09-26 04:08:35,862 [I] --------- Verifying version history table ---------
2026-09-26 04:08:35,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:08:36,108 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 04:08:36,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:08:36,360 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 04:08:36,361 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 04:08:36,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 04:08:36.626 UTC [11691] WARNING: there is no transaction in progress
2026-09-26 04:08:36,632 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 04:08:36,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:08:36,899 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 04:08:36,900 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 04:08:36,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 04:08:37,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 04:08:37,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xf24.c.gcp-upstream.internal','N') ;"
2026-09-26 04:08:37,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 04:08:37,868 [I] Table x_portal_user does not exist in database ranger
2026-09-26 04:08:37,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 04:08:38,102 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 04:08:38,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 04:08:38,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 04:08:38,598 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 04:08:38,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 04:08:39.637 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.655 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.674 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.691 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.710 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.728 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.804 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.811 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.828 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.839 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.850 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.867 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.878 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.885 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.893 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:39.901 UTC [11799] WARNING: there is no transaction in progress
2026-09-26 04:08:40,618 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 04:08:40,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 04:08:40,858 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 04:08:40,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 04:08:41,154 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 04:08:41,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 04:08:41,409 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 04:08:41,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 04:08:41,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 04:08:41,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xf24.c.gcp-upstream.internal';"
2026-09-26 04:08:42,163 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 04:08:42,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 04:08:42,427 [I] DB_PATCHES have already been applied
I20260926 04:08:42.433054 855 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrg1Sr7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 04:08:42.433120 855 mini_ranger.cc:193] Using host: 127.0.213.212
I20260926 04:08:42.436157 855 mini_ranger.cc:227] Ranger admin URL: http://127.0.213.212:38865
Sep 26, 2026 4:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 04:08:43.550611 855 mini_ranger.cc:160] Time spent starting Ranger: real 8.791s user 0.000s sys 0.009s
Sep 26, 2026 4:09:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:09:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:09:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:09:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:09:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 04:09:06.923183 855 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/rangerkms_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/rangerkms_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.213.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.213.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 04:09:07 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395747, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 04:09:07.014267 855 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 04:09:07.024051 855 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 04:09:08.036172 855 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 04:09:08.036276 855 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms
I20260926 04:09:08.036299 855 mini_ranger_kms.cc:227] Using postgres at 127.0.213.212:44479
2026-09-26 04:09:08,265 [I] DB FLAVOR :POSTGRES
2026-09-26 04:09:08,266 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 04:09:08,266 [I] Checking connection
2026-09-26 04:09:08,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 04:09:08,568 [I] connection success
2026-09-26 04:09:08,568 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 04:09:08,568 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 04:09:08,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 04:09:08,825 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 04:09:08,825 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 04:09:08,825 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 04:09:08,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:44479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 04:09:09,115 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 04:09:09.669363 855 mini_ranger_kms.cc:310] Created kms service
I20260926 04:09:09.840828 855 mini_ranger_kms.cc:326] Created kudu user
I20260926 04:09:09.886164 855 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 04:09:09 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395747, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:09:10.352535 855 mini_ranger_kms.cc:384] Added ranger policy
I20260926 04:09:10.352670 855 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 04:09:10.352692 855 mini_ranger_kms.cc:240] Using host: 127.0.213.212
I20260926 04:09:10.355867 855 mini_ranger_kms.cc:276] Ranger KMS PID: 11980
I20260926 04:09:10.355993 855 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.213.212:42183
Sep 26, 2026 4:09:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:09:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:09:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:09:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:09:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:09:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 04:09:12.590162 855 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.576s user 0.003s sys 0.203s
I20260926 04:09:12.590276 855 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 04:09:12.590361 855 mini_ranger_kms.cc:401] 127.0.213.212:42183/kms/v1/keys
Sep 26 04:09:12 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.213.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 04:09:13 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.213.212@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
Sep 26 04:09:13 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.213.212@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
Sep 26 04:09:15 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:09:15.936437 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45335
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45335
--ranger_config_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:42183/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:09:16.022643 12030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:09:16.022864 12030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:09:16.022887 12030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:09:16.024159 12030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:09:16.024202 12030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:09:16.024217 12030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:09:16.024231 12030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:09:16.024243 12030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:09:16.025784 12030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.213.212:42183/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45335
--ranger_config_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:09:16.025976 12030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:09:16.026149 12030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:09:16.031620 12036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:09:16.031625 12038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:16.031668 12030 server_base.cc:1034] running on GCE node
W20260926 04:09:16.031633 12035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:16.032284 12030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:09:16.032603 12030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:09:16.033787 12030 hybrid_clock.cc:648] HybridClock initialized: now 1790395756033768 us; error 56 us; skew 500 ppm
Sep 26 04:09:16 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:09:16.043334 12030 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:09:16.043845 12030 webserver.cc:460] Webserver started at http://127.0.213.254:35533/ using document root <none> and password file <none>
I20260926 04:09:16.044029 12030 fs_manager.cc:359] Metadata directory not provided
I20260926 04:09:16.044124 12030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:09:16.044236 12030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:09:16 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:09:16.108525 12030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "731ffd0b50374eaeacdb0bbffc7f4595"
format_stamp: "Formatted at 2026-09-26 04:09:16 on dist-test-slave-xf24"
server_key: "4dc1759d0f515a770e7f8017d7895c83"
server_key_iv: "b7a0c59c0e95bde4b56f5d37d7eefc61"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:16.108951 12030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "731ffd0b50374eaeacdb0bbffc7f4595"
format_stamp: "Formatted at 2026-09-26 04:09:16 on dist-test-slave-xf24"
server_key: "4dc1759d0f515a770e7f8017d7895c83"
server_key_iv: "b7a0c59c0e95bde4b56f5d37d7eefc61"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:16.110464 12030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 04:09:16.126746 12045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:16.126994 12030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:09:16.127081 12030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "731ffd0b50374eaeacdb0bbffc7f4595"
format_stamp: "Formatted at 2026-09-26 04:09:16 on dist-test-slave-xf24"
server_key: "4dc1759d0f515a770e7f8017d7895c83"
server_key_iv: "b7a0c59c0e95bde4b56f5d37d7eefc61"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:16.127188 12030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:09:16.137060 12030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:09:16.139194 12030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:09:16.139377 12030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:09:16.144918 12030 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45335
I20260926 04:09:16.144936 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45335 every 8 connection(s)
I20260926 04:09:16.145364 12030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:09:16.146188 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:09:16.149528 12098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595: Bootstrap starting.
I20260926 04:09:16.150278 12098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595: Neither blocks nor log segments found. Creating new log.
I20260926 04:09:16.150615 12098 log.cc:824] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595: Log is configured to *not* fsync() on all Append() calls
I20260926 04:09:16.151521 12098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595: No bootstrap required, opened a new log
I20260926 04:09:16.153112 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 12030
I20260926 04:09:16.153292 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:09:16.153491 855 external_mini_cluster.cc:1428] Setting key 67eb5fb7257b705d2455aa3dfda376a9
I20260926 04:09:16.155309 12098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45335 } }
I20260926 04:09:16.155542 12098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:09:16.155638 12098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731ffd0b50374eaeacdb0bbffc7f4595, State: Initialized, Role: FOLLOWER
I20260926 04:09:16.155814 12098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45335 } }
I20260926 04:09:16.155895 12098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:09:16.155926 12098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:09:16.155990 12098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:09:16.156872 12098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45335 } }
I20260926 04:09:16.156991 12098 leader_election.cc:302] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 731ffd0b50374eaeacdb0bbffc7f4595; no voters:
I20260926 04:09:16.157210 12098 leader_election.cc:288] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:09:16.157262 12103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:09:16.157495 12103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [term 1 LEADER]: Becoming Leader. State: Replica: 731ffd0b50374eaeacdb0bbffc7f4595, State: Running, Role: LEADER
I20260926 04:09:16.157572 12098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:09:16.157626 12103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45335 } }
I20260926 04:09:16.159008 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45335 } } }
I20260926 04:09:16.159241 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [sys.catalog]: This master's current role is: LEADER
I20260926 04:09:16.159301 12098 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 04:09:16.159415 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 731ffd0b50374eaeacdb0bbffc7f4595. Latest consensus state: current_term: 1 leader_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731ffd0b50374eaeacdb0bbffc7f4595" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45335 } } }
I20260926 04:09:16.159476 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595 [sys.catalog]: This master's current role is: LEADER
I20260926 04:09:16.159832 12117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 04:09:16 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:09:16.166594 12117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:09:16.168695 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:16.154056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.0.213.254:45335)
0926 04:09:16.154285 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:16.154297 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:16.154332 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:09:16.155330 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:16.155340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:16.155402 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:16.155594 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:09:16.156561 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:16.157847 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:16.158855 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:16.159542 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:16.159655 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:16.163326 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:09:16.163382 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:09:16.163383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:09:16.163477 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:09:16.166261 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:16.166775 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:16.166779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:16.166780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:16.166907 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:16.167132 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:16.167133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:16.167133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:16.167527 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:09:16.167689 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:16.168298 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 04:09:16.168459 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20260926 04:09:16.171689 12117 catalog_manager.cc:1347] Generated new cluster ID: 2dcc7d2b2b0247c88800ae9a9a104ded
I20260926 04:09:16.171773 12117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:09:16.181025 12117 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:09:16.182281 12117 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:09:16.204331 12117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595: Generated new TSK 0
I20260926 04:09:16.204689 12117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:09:16.971593 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:42183/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45335
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:09:17.065948 12143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:09:17.066190 12143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:09:17.066233 12143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:09:17.067436 12143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:09:17.067519 12143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:09:17.067566 12143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:09:17.069521 12143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.213.212:42183/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:09:17.069840 12143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:09:17.070091 12143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:09:17.075541 12151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:17.075706 12143 server_base.cc:1034] running on GCE node
W20260926 04:09:17.075547 12148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:09:17.075973 12149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:17.076258 12143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:09:17.076582 12143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:09:17.077798 12143 hybrid_clock.cc:648] HybridClock initialized: now 1790395757077776 us; error 39 us; skew 500 ppm
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:09:17.080633 12143 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:09:17.081105 12143 webserver.cc:460] Webserver started at http://127.0.213.193:37695/ using document root <none> and password file <none>
I20260926 04:09:17.081372 12143 fs_manager.cc:359] Metadata directory not provided
I20260926 04:09:17.081447 12143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:09:17.081611 12143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:09:17.097344 12143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "04a0ad299dc24e3ea2c0550a4213d4c2"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "97ed05693165e88f0eff6efbc27df24e"
server_key_iv: "09691fe3e8f6ff282af6596339da97b8"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.098048 12143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "04a0ad299dc24e3ea2c0550a4213d4c2"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "97ed05693165e88f0eff6efbc27df24e"
server_key_iv: "09691fe3e8f6ff282af6596339da97b8"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.100544 12143 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:09:17.115569 12158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.115886 12143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:09:17.115984 12143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "04a0ad299dc24e3ea2c0550a4213d4c2"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "97ed05693165e88f0eff6efbc27df24e"
server_key_iv: "09691fe3e8f6ff282af6596339da97b8"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.116137 12143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:09:17.143416 12143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:09:17.145843 12143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:09:17.146054 12143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:09:17.146335 12143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:09:17.146785 12143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:09:17.146893 12143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.146993 12143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:09:17.147066 12143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.153457 12143 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:38087
I20260926 04:09:17.153574 12273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:38087 every 8 connection(s)
I20260926 04:09:17.153901 12143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:09:17.163426 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 12143
I20260926 04:09:17.163542 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:09:17.163748 855 external_mini_cluster.cc:1428] Setting key bdc72f431b4fc2a524d544d1e857d864
I20260926 04:09:17.166044 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.154742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45335 (local address 127.0.213.193:60773)
0926 04:09:17.154969 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.155019 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.155668 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.156174 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.156178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.156450 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:09:17.156749 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.156756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.157810 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.157811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.158925 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.158929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.159435 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.159436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.159978 (+ 542us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.160659 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:09:17.160669 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:09:17.161988 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:09:17.164167 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:09:17.164176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:09:17.164179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:09:17.164530 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:09:17.164774 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:09:17.164776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:09:17.164776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:09:17.164927 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:09:17.165232 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:09:17.165237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:09:17.165427 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.165538 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.165732 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260926 04:09:17.166456 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.154790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:60773 (local address 127.0.213.254:45335)
0926 04:09:17.155878 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.155881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.155890 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:09:17.155919 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.155920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.155934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.156005 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:09:17.156887 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.157681 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.159094 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.159329 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.159425 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.162108 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:09:17.162110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:09:17.162110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:09:17.162119 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:09:17.163939 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.164615 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.164616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.164616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.164685 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.164995 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.164996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.164996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.165146 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:09:17.165206 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.166277 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.166375 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":81,"threads_started":1}
I20260926 04:09:17.167037 12274 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45335
I20260926 04:09:17.167162 12274 heartbeater.cc:464] Registering TS with master...
I20260926 04:09:17.167359 12274 heartbeater.cc:511] Master 127.0.213.254:45335 requested a full tablet report, sending...
I20260926 04:09:17.167920 12062 ts_manager.cc:194] Registered new tserver with Master: 04a0ad299dc24e3ea2c0550a4213d4c2 (127.0.213.193:38087)
I20260926 04:09:17.168888 12062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:60773
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:09:17.195873 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:42183/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45335
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 04:09:17.321266 12280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:09:17.321672 12280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:09:17.321790 12280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:09:17.323825 12280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:09:17.323995 12280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:09:17.324156 12280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:09:17.326917 12280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.213.212:42183/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:09:17.327462 12280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:09:17.327919 12280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:09:17.335000 12290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:09:17.335356 12288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:17.335753 12280 server_base.cc:1034] running on GCE node
W20260926 04:09:17.335942 12287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:17.336247 12280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:09:17.336534 12280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:09:17.337728 12280 hybrid_clock.cc:648] HybridClock initialized: now 1790395757337695 us; error 36 us; skew 500 ppm
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:09:17.341176 12280 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:09:17.341845 12280 webserver.cc:460] Webserver started at http://127.0.213.194:39093/ using document root <none> and password file <none>
I20260926 04:09:17.342213 12280 fs_manager.cc:359] Metadata directory not provided
I20260926 04:09:17.342314 12280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:09:17.342491 12280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:09:17.361800 12280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "b2e670a7c2c04b99a49ecd1e336adc21"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "fa40481f0232d8d813c9a9e8860d114b"
server_key_iv: "55f6e5493a1545b05b81f515d18b132a"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.362417 12280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "b2e670a7c2c04b99a49ecd1e336adc21"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "fa40481f0232d8d813c9a9e8860d114b"
server_key_iv: "55f6e5493a1545b05b81f515d18b132a"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.365599 12280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 04:09:17.388214 12297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.388526 12280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:09:17.388659 12280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "b2e670a7c2c04b99a49ecd1e336adc21"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "fa40481f0232d8d813c9a9e8860d114b"
server_key_iv: "55f6e5493a1545b05b81f515d18b132a"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.388785 12280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:09:17.403561 12280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:09:17.406340 12280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:09:17.406575 12280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:09:17.407183 12280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:09:17.407658 12280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:09:17.407770 12280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.407846 12280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:09:17.407902 12280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.414883 12280 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:39317
I20260926 04:09:17.415966 12280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:09:17.419339 12411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:39317 every 8 connection(s)
I20260926 04:09:17.422562 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 12280
I20260926 04:09:17.422669 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:09:17.422845 855 external_mini_cluster.cc:1428] Setting key d06a62352818f2f239e383c2ac273b61
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:09:17.435154 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.420432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:53997 (local address 127.0.213.254:45335)
0926 04:09:17.420487 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.420490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.420671 (+ 181us) server_negotiation.cc:407] Connection header received
0926 04:09:17.421426 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.421428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.421444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.421477 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:09:17.425270 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.426085 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.427822 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.428209 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.428261 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.431359 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:09:17.431361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:09:17.431361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:09:17.431371 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:09:17.433385 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.433945 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.433946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.433947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.434011 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.434329 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.434330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.434331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.434485 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:09:17.434527 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.434932 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.435033 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:09:17.436002 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.420117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45335 (local address 127.0.213.194:53997)
0926 04:09:17.420528 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.420583 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.421295 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.421671 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.421676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.421992 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:09:17.422291 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.422297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.426224 (+ 3927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.426225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.427611 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.427615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.428399 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.428400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.428562 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.429251 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:09:17.429262 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:09:17.431211 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:09:17.433504 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:09:17.433514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:09:17.433516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:09:17.433847 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:09:17.434082 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:09:17.434083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:09:17.434083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:09:17.434241 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:09:17.434572 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:09:17.434578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:09:17.434773 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.435547 (+ 774us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.435755 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":96,"threads_started":1}
I20260926 04:09:17.436714 12412 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45335
I20260926 04:09:17.436810 12412 heartbeater.cc:464] Registering TS with master...
I20260926 04:09:17.436998 12412 heartbeater.cc:511] Master 127.0.213.254:45335 requested a full tablet report, sending...
I20260926 04:09:17.437592 12062 ts_manager.cc:194] Registered new tserver with Master: b2e670a7c2c04b99a49ecd1e336adc21 (127.0.213.194:39317)
I20260926 04:09:17.438056 12062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:53997
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:09:17.476987 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:42183/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45335
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:09:17.597170 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:09:17.597481 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:09:17.597522 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:09:17.599478 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:09:17.599593 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:09:17.599669 12417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:09:17.602056 12417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.213.212:42183/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:09:17.602345 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:09:17.602679 12417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:09:17.609997 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:09:17.610392 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:09:17.610535 12422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:09:17.611447 12417 server_base.cc:1034] running on GCE node
I20260926 04:09:17.611970 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:09:17.612340 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:09:17.613543 12417 hybrid_clock.cc:648] HybridClock initialized: now 1790395757613521 us; error 41 us; skew 500 ppm
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:09:17.616595 12417 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:09:17.617148 12417 webserver.cc:460] Webserver started at http://127.0.213.195:35115/ using document root <none> and password file <none>
I20260926 04:09:17.617515 12417 fs_manager.cc:359] Metadata directory not provided
I20260926 04:09:17.617624 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:09:17.617926 12417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:09:17.633263 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "cde4d714e6bc4bccb6cbfb78481e896a"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "2b535796b50df735bc02eeaae0cc1080"
server_key_iv: "dd6070f02d2fa94e170e610c33da9008"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.633910 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "cde4d714e6bc4bccb6cbfb78481e896a"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "2b535796b50df735bc02eeaae0cc1080"
server_key_iv: "dd6070f02d2fa94e170e610c33da9008"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.636164 12417 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:09:17.670394 12432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.670723 12417 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 04:09:17.670822 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "cde4d714e6bc4bccb6cbfb78481e896a"
format_stamp: "Formatted at 2026-09-26 04:09:17 on dist-test-slave-xf24"
server_key: "2b535796b50df735bc02eeaae0cc1080"
server_key_iv: "dd6070f02d2fa94e170e610c33da9008"
server_key_version: "kuduclusterkey@0"
I20260926 04:09:17.670923 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:09:17.689402 12417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:09:17.692123 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:09:17.692512 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:09:17.692871 12417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:09:17.693316 12417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:09:17.693375 12417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.693410 12417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:09:17.693488 12417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:09:17.702486 12417 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:39181
I20260926 04:09:17.702981 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:09:17.705523 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 12417
I20260926 04:09:17.705631 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:09:17.705895 855 external_mini_cluster.cc:1428] Setting key 01797dbc9f27dd1f9628c480cae63aaa
I20260926 04:09:17.704578 12545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:39181 every 8 connection(s)
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:09:17.715198 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.704196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:55827 (local address 127.0.213.254:45335)
0926 04:09:17.704265 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.704268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.704957 (+ 689us) server_negotiation.cc:407] Connection header received
0926 04:09:17.705844 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.705846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.705859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.705892 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:09:17.706889 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.707668 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.709136 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.709366 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.709459 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.712049 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:09:17.712050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:09:17.712073 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:09:17.712083 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:09:17.713575 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.714129 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.714130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.714131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.714200 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.714490 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.714491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.714491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.714623 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:09:17.714704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.715016 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.715104 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:09:17.715415 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.703948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45335 (local address 127.0.213.195:55827)
0926 04:09:17.704811 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.704858 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.705573 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.706140 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.706144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.706401 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:09:17.706684 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.706690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.707801 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.707802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.709033 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.709036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.709470 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.709472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.709605 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.710221 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:09:17.710267 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:09:17.711935 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:09:17.713693 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:09:17.713704 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:09:17.713707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:09:17.714039 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:09:17.714272 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:09:17.714273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:09:17.714274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:09:17.714410 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:09:17.714695 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:09:17.714702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:09:17.714869 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.714985 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.715150 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":268,"threads_started":1}
I20260926 04:09:17.716322 12546 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45335
I20260926 04:09:17.716455 12546 heartbeater.cc:464] Registering TS with master...
I20260926 04:09:17.716653 12546 heartbeater.cc:511] Master 127.0.213.254:45335 requested a full tablet report, sending...
I20260926 04:09:17.717118 12062 ts_manager.cc:194] Registered new tserver with Master: cde4d714e6bc4bccb6cbfb78481e896a (127.0.213.195:39181)
I20260926 04:09:17.717530 12062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:55827
I20260926 04:09:17.718097 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:09:17.730701 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.721607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.0.213.254:45335)
0926 04:09:17.721692 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.721694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.722004 (+ 310us) server_negotiation.cc:407] Connection header received
0926 04:09:17.722087 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.722089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.722100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.722147 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:09:17.722739 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.723246 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.724015 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.724322 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.724396 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.726821 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:09:17.726823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:09:17.726823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:09:17.726832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:09:17.728367 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.729611 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.729613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.729614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.729683 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:09:17.729953 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:09:17.729954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:09:17.729955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:09:17.730111 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:09:17.730232 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.730350 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.730442 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 04:09:17.732760 12062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:09:17.734803 12062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:09:17.748739 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.738046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:38087 (local address 127.0.0.1:38062)
0926 04:09:17.739273 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.739283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.739333 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.739822 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.739825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.739827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:09:17.739900 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.739903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.744624 (+ 4721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.744625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.745732 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.745735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.748389 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.748391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.748491 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.748508 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.748579 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.748640 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":68,"threads_started":1}
I20260926 04:09:17.748953 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.737809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:39181 (local address 127.0.0.1:54226)
0926 04:09:17.738844 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.738853 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.738890 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.739879 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.739879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.739881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:09:17.739907 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.739908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.744980 (+ 5072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.744981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.746391 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.746394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.748674 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.748676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.748760 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.748765 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.748820 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.748867 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260926 04:09:17.749616 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.737809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:39317 (local address 127.0.0.1:34500)
0926 04:09:17.738601 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.738646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.738725 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.739977 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.739977 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.739978 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:09:17.740003 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.740004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.743125 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.743126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.744520 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.744524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.749019 (+ 4495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.749020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.749430 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.749435 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.749491 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.749537 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":92,"threads_started":1}
I20260926 04:09:17.749652 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.738188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.213.193:38087)
0926 04:09:17.738305 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.738311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.739665 (+ 1354us) server_negotiation.cc:407] Connection header received
0926 04:09:17.739704 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.739706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.739718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.739810 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:09:17.739984 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.744512 (+ 4528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.747391 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.748199 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.748291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.748350 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.749517 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.749572 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260926 04:09:17.750344 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.737939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.0.213.194:39317)
0926 04:09:17.739014 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.739019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.739031 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:09:17.739073 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.739075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.739091 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.739168 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:09:17.740410 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.741518 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.746529 (+ 5011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.747405 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.747481 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.747568 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.750189 (+ 2621us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.750249 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":100,"threads_started":1}
I20260926 04:09:17.750679 12206 tablet_service.cc:1467] Processing CreateTablet for tablet 92bc2d47adba408a8c04f348c77ee223 (DEFAULT_TABLE table=test-table [id=defe80842c1540c4a0aa7b569ede65c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:09:17.751086 12206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bc2d47adba408a8c04f348c77ee223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:09:17.751236 12346 tablet_service.cc:1467] Processing CreateTablet for tablet 92bc2d47adba408a8c04f348c77ee223 (DEFAULT_TABLE table=test-table [id=defe80842c1540c4a0aa7b569ede65c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:09:17.751578 12346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bc2d47adba408a8c04f348c77ee223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:09:17.755151 12565 tablet_bootstrap.cc:492] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Bootstrap starting.
I20260926 04:09:17.755771 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.738270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.213.195:39181)
0926 04:09:17.739443 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.739447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.739456 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:09:17.739496 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.739498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.739514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.739588 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:09:17.741618 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.744890 (+ 3272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.747627 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.748480 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.748558 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.748631 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.755572 (+ 6941us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.755678 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":109,"threads_started":1}
I20260926 04:09:17.756278 12565 tablet_bootstrap.cc:654] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Neither blocks nor log segments found. Creating new log.
I20260926 04:09:17.756644 12565 log.cc:824] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Log is configured to *not* fsync() on all Append() calls
I20260926 04:09:17.756702 12462 tablet_service.cc:1467] Processing CreateTablet for tablet 92bc2d47adba408a8c04f348c77ee223 (DEFAULT_TABLE table=test-table [id=defe80842c1540c4a0aa7b569ede65c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:09:17.757068 12462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bc2d47adba408a8c04f348c77ee223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:09:17.757089 12564 tablet_bootstrap.cc:492] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: Bootstrap starting.
I20260926 04:09:17.757766 12564 tablet_bootstrap.cc:654] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: Neither blocks nor log segments found. Creating new log.
I20260926 04:09:17.758651 12564 log.cc:824] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: Log is configured to *not* fsync() on all Append() calls
I20260926 04:09:17.759493 12565 tablet_bootstrap.cc:492] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: No bootstrap required, opened a new log
I20260926 04:09:17.759567 12565 ts_tablet_manager.cc:1397] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 04:09:17.761788 12565 raft_consensus.cc:348] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.762106 12565 raft_consensus.cc:374] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:09:17.762212 12565 raft_consensus.cc:729] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e670a7c2c04b99a49ecd1e336adc21, State: Initialized, Role: FOLLOWER
I20260926 04:09:17.762436 12565 consensus_queue.cc:260] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.762791 12565 ts_tablet_manager.cc:1428] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:09:17.762943 12412 heartbeater.cc:503] Master 127.0.213.254:45335 was elected leader, sending a full tablet report...
I20260926 04:09:17.763916 12564 tablet_bootstrap.cc:492] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: No bootstrap required, opened a new log
I20260926 04:09:17.764036 12564 ts_tablet_manager.cc:1397] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 04:09:17.766803 12564 raft_consensus.cc:348] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.767058 12564 raft_consensus.cc:374] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:09:17.767130 12564 raft_consensus.cc:729] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a0ad299dc24e3ea2c0550a4213d4c2, State: Initialized, Role: FOLLOWER
I20260926 04:09:17.767331 12564 consensus_queue.cc:260] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.767670 12564 ts_tablet_manager.cc:1428] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 04:09:17.767848 12274 heartbeater.cc:503] Master 127.0.213.254:45335 was elected leader, sending a full tablet report...
I20260926 04:09:17.768625 12568 tablet_bootstrap.cc:492] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Bootstrap starting.
I20260926 04:09:17.769747 12568 tablet_bootstrap.cc:654] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Neither blocks nor log segments found. Creating new log.
I20260926 04:09:17.770068 12568 log.cc:824] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:09:17.771416 12568 tablet_bootstrap.cc:492] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: No bootstrap required, opened a new log
I20260926 04:09:17.771531 12568 ts_tablet_manager.cc:1397] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:09:17.773556 12568 raft_consensus.cc:348] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.773788 12568 raft_consensus.cc:374] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:09:17.773833 12568 raft_consensus.cc:729] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde4d714e6bc4bccb6cbfb78481e896a, State: Initialized, Role: FOLLOWER
I20260926 04:09:17.774039 12568 consensus_queue.cc:260] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.774376 12568 ts_tablet_manager.cc:1428] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:09:17.774528 12546 heartbeater.cc:503] Master 127.0.213.254:45335 was elected leader, sending a full tablet report...
Sep 26 04:09:17 dist-test-slave-xf24 krb5kdc[11586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
W20260926 04:09:17.904874 12275 tablet.cc:2369] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:09:17.927210 12570 raft_consensus.cc:482] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:09:17.927376 12570 raft_consensus.cc:504] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.927759 12570 leader_election.cc:288] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde4d714e6bc4bccb6cbfb78481e896a (127.0.213.195:39181), b2e670a7c2c04b99a49ecd1e336adc21 (127.0.213.194:39317)
I20260926 04:09:17.932597 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.928209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:39713 (local address 127.0.213.194:39317)
0926 04:09:17.928282 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.928284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.928779 (+ 495us) server_negotiation.cc:407] Connection header received
0926 04:09:17.928888 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.928890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.928904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.928933 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:09:17.929194 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.929934 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.931607 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.932179 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.932229 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.932239 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.932456 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.932501 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 04:09:17.932796 12413 tablet.cc:2369] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:09:17.933382 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.927991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:39317 (local address 127.0.213.193:39713)
0926 04:09:17.928694 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.928705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.928812 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.929001 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.929002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.929015 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:09:17.929108 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.929110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.930088 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.930089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.931479 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.931484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.932302 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.932303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.932375 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.932380 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.933242 (+ 862us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.933298 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":110,"threads_started":1}
I20260926 04:09:17.934020 12366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bc2d47adba408a8c04f348c77ee223" candidate_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" is_pre_election: true
I20260926 04:09:17.934185 12366 raft_consensus.cc:2454] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a0ad299dc24e3ea2c0550a4213d4c2 in term 0.
I20260926 04:09:17.934459 12161 leader_election.cc:302] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a0ad299dc24e3ea2c0550a4213d4c2, b2e670a7c2c04b99a49ecd1e336adc21; no voters:
I20260926 04:09:17.934617 12570 raft_consensus.cc:2781] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:09:17.934659 12570 raft_consensus.cc:482] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:09:17.934677 12570 raft_consensus.cc:3037] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:09:17.935812 12570 raft_consensus.cc:504] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.935971 12570 leader_election.cc:288] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [CANDIDATE]: Term 1 election: Requested vote from peers cde4d714e6bc4bccb6cbfb78481e896a (127.0.213.195:39181), b2e670a7c2c04b99a49ecd1e336adc21 (127.0.213.194:39317)
I20260926 04:09:17.936435 12366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bc2d47adba408a8c04f348c77ee223" candidate_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e670a7c2c04b99a49ecd1e336adc21"
I20260926 04:09:17.936519 12366 raft_consensus.cc:3037] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:09:17.937378 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.928476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:34555 (local address 127.0.213.195:39181)
0926 04:09:17.928529 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.928531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.933577 (+ 5046us) server_negotiation.cc:407] Connection header received
0926 04:09:17.933693 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.933694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.933704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.933748 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:09:17.933962 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.934742 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.936393 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.936973 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.937027 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.937037 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.937263 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.937307 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 04:09:17.937410 12366 raft_consensus.cc:2454] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a0ad299dc24e3ea2c0550a4213d4c2 in term 1.
I20260926 04:09:17.937808 12161 leader_election.cc:302] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a0ad299dc24e3ea2c0550a4213d4c2, b2e670a7c2c04b99a49ecd1e336adc21; no voters:
I20260926 04:09:17.937945 12570 raft_consensus.cc:2781] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:09:17.938174 12570 raft_consensus.cc:686] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [term 1 LEADER]: Becoming Leader. State: Replica: 04a0ad299dc24e3ea2c0550a4213d4c2, State: Running, Role: LEADER
I20260926 04:09:17.938295 12570 consensus_queue.cc:237] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } }
I20260926 04:09:17.939208 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.928399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:39181 (local address 127.0.213.193:34555)
0926 04:09:17.933488 (+ 5089us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:17.933503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:09:17.933613 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:09:17.933819 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:09:17.933820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:09:17.933822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:09:17.933887 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.933888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.934907 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.934908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.936272 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:09:17.936276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.937100 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.937101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:09:17.937177 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.937181 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:09:17.937493 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 04:09:17.937542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5065,"thread_start_us":101,"threads_started":1}
I20260926 04:09:17.939178 12061 catalog_manager.cc:5495] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04a0ad299dc24e3ea2c0550a4213d4c2 (127.0.213.193). New cstate: current_term: 1 leader_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 38087 } health_report { overall_health: HEALTHY } } }
I20260926 04:09:17.939756 12500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bc2d47adba408a8c04f348c77ee223" candidate_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" is_pre_election: true
I20260926 04:09:17.939910 12500 raft_consensus.cc:2454] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a0ad299dc24e3ea2c0550a4213d4c2 in term 0.
I20260926 04:09:17.940018 12499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bc2d47adba408a8c04f348c77ee223" candidate_uuid: "04a0ad299dc24e3ea2c0550a4213d4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde4d714e6bc4bccb6cbfb78481e896a"
I20260926 04:09:17.940091 12499 raft_consensus.cc:3037] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:09:17.940938 12499 raft_consensus.cc:2454] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a0ad299dc24e3ea2c0550a4213d4c2 in term 1.
W20260926 04:09:17.954741 12547 tablet.cc:2369] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:09:17.971807 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.968935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.0.213.193:38087)
0926 04:09:17.969007 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 04:09:17.969009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:09:17.969019 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:09:17.969061 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:09:17.969062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:09:17.969087 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:09:17.969164 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:09:17.969311 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.970027 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.970896 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:09:17.971126 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:09:17.971216 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:09:17.971417 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:09:17.971535 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:09:17.971640 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 04:09:17.971659 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 04:09:17.971683 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 04:09:17.973752 12499 raft_consensus.cc:1261] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 1 FOLLOWER]: Refusing update from remote peer 04a0ad299dc24e3ea2c0550a4213d4c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:09:17.973789 12366 raft_consensus.cc:1261] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 1 FOLLOWER]: Refusing update from remote peer 04a0ad299dc24e3ea2c0550a4213d4c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:09:17.974099 12570 consensus_queue.cc:1035] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e670a7c2c04b99a49ecd1e336adc21" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:09:17.974395 12570 consensus_queue.cc:1035] T 92bc2d47adba408a8c04f348c77ee223 P 04a0ad299dc24e3ea2c0550a4213d4c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde4d714e6bc4bccb6cbfb78481e896a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:09:17.979151 12580 mvcc.cc:205] Tried to move back new op lower bound from 7333461024657563648 to 7333461024515805184. Current Snapshot: MvccSnapshot[applied={T|T < 7333461024657563648}]
I20260926 04:09:17.977907 12578 mvcc.cc:205] Tried to move back new op lower bound from 7333461024657563648 to 7333461024515805184. Current Snapshot: MvccSnapshot[applied={T|T < 7333461024657563648}]
I20260926 04:09:17.982350 12062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:09:17.982489 12062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:09:17.983347 12062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 731ffd0b50374eaeacdb0bbffc7f4595: Sending DeleteTablet for 3 replicas of tablet 92bc2d47adba408a8c04f348c77ee223
I20260926 04:09:17.983814 12346 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bc2d47adba408a8c04f348c77ee223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:09:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:34500
I20260926 04:09:17.983873 12462 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bc2d47adba408a8c04f348c77ee223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:09:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:54226
I20260926 04:09:17.984115 12206 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bc2d47adba408a8c04f348c77ee223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:09:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:38062
I20260926 04:09:17.984370 12587 tablet_replica.cc:330] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: stopping tablet replica
I20260926 04:09:17.984576 12587 raft_consensus.cc:2229] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:09:17.984660 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 12143
I20260926 04:09:17.984773 12587 raft_consensus.cc:2258] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:09:17.986121 12587 ts_tablet_manager.cc:1905] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:09:17.987960 12587 ts_tablet_manager.cc:1918] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:09:17.988102 12587 log.cc:1196] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/92bc2d47adba408a8c04f348c77ee223
I20260926 04:09:17.988385 12587 ts_tablet_manager.cc:1939] T 92bc2d47adba408a8c04f348c77ee223 P b2e670a7c2c04b99a49ecd1e336adc21: Deleting consensus metadata
I20260926 04:09:17.988919 12048 catalog_manager.cc:4841] TS b2e670a7c2c04b99a49ecd1e336adc21 (127.0.213.194:39317): tablet 92bc2d47adba408a8c04f348c77ee223 (table test-table [id=defe80842c1540c4a0aa7b569ede65c3]) successfully deleted
I20260926 04:09:17.993140 12588 tablet_replica.cc:330] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: stopping tablet replica
I20260926 04:09:17.993445 12588 raft_consensus.cc:2229] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:09:17.993671 12588 raft_consensus.cc:2258] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:09:17.994383 12588 ts_tablet_manager.cc:1905] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 04:09:17.996536 12047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:38087 (error 108)
I20260926 04:09:17.997004 12588 ts_tablet_manager.cc:1918] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:09:17.997174 12588 log.cc:1196] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/92bc2d47adba408a8c04f348c77ee223
I20260926 04:09:17.997244 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 12280
I20260926 04:09:17.998721 12588 ts_tablet_manager.cc:1939] T 92bc2d47adba408a8c04f348c77ee223 P cde4d714e6bc4bccb6cbfb78481e896a: Deleting consensus metadata
I20260926 04:09:17.999275 12047 catalog_manager.cc:4841] TS cde4d714e6bc4bccb6cbfb78481e896a (127.0.213.195:39181): tablet 92bc2d47adba408a8c04f348c77ee223 (table test-table [id=defe80842c1540c4a0aa7b569ede65c3]) successfully deleted
I20260926 04:09:18.003983 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:09:17.999454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:38087 (local address 127.0.0.1:38082)
0926 04:09:18.003857 (+ 4403us) negotiation.cc:107] Waiting for socket to connect
0926 04:09:18.003932 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:38087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4367}
W20260926 04:09:18.004352 12047 catalog_manager.cc:4584] TS 04a0ad299dc24e3ea2c0550a4213d4c2 (127.0.213.193:38087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92bc2d47adba408a8c04f348c77ee223: Network error: Client connection negotiation failed: client connection to 127.0.213.193:38087: connect: Connection refused (error 111)
I20260926 04:09:18.004429 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 12417
I20260926 04:09:18.009785 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 12030
I20260926 04:09:18.016458 855 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 04:09:18.362315 855 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 04:09:18.362438 855 mini_ranger.cc:66] Stopping Ranger...
I20260926 04:09:18.707455 855 mini_ranger.cc:68] Stopped Ranger
2026-09-26 04:09:18.707 UTC [11616] LOG: received smart shutdown request
2026-09-26 04:09:18.710 UTC [11616] LOG: background worker "logical replication launcher" (PID 11625) exited with exit code 1
2026-09-26 04:09:18.711 UTC [11620] LOG: shutting down
2026-09-26 04:09:18.901 UTC [11616] LOG: database system is shut down
2026-09-26T04:09:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49283 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:09:19 dist-test-slave-xf24 krb5kdc[12599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:09:19 dist-test-slave-xf24 krb5kdc[12599](info): set up 2 sockets
Sep 26 04:09:19 dist-test-slave-xf24 krb5kdc[12599](info): commencing operation
krb5kdc: starting...
W20260926 04:09:20.076578 855 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:09:20 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:09:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/rangeradmin_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/rangeradmin_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/rangerlookup_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/rangerlookup_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
I20260926 04:09:20.251000 855 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrg1Sr7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/postgres -l logfile start
2026-09-26 04:09:23.166 UTC [12629] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 04:09:23.166 UTC [12629] LOG: listening on IPv4 address "127.0.213.212", port 50411
2026-09-26 04:09:23.171 UTC [12629] LOG: listening on Unix socket "/tmp/.s.PGSQL.50411"
2026-09-26 04:09:23.208 UTC [12632] LOG: database system was shut down at 2026-09-26 04:09:21 UTC
2026-09-26 04:09:23.213 UTC [12629] LOG: database system is ready to accept connections
I20260926 04:09:24.163916 855 mini_postgres.cc:96] Postgres bound to 50411
2026-09-26 04:09:24.170 UTC [12640] FATAL: database "slave" does not exist
127.0.213.212:50411 - accepting connections
I20260926 04:09:24.170995 855 mini_ranger.cc:161] Starting Ranger...
I20260926 04:09:24.186743 855 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 04:09:24.739662 855 mini_ranger.cc:87] Created ranger Postgres database
I20260926 04:09:24.739806 855 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-admin
2026-09-26 04:09:25,035 [I] DB FLAVOR :POSTGRES
2026-09-26 04:09:25,036 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 04:09:25,036 [I] Checking connection..
2026-09-26 04:09:25,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 04:09:25,291 [I] Checking connection passed.
2026-09-26 04:09:25,291 [I] --------- Verifying version history table ---------
2026-09-26 04:09:25,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:09:25,558 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 04:09:25,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:09:25,828 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 04:09:25,828 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 04:09:25,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 04:09:26.076 UTC [12704] WARNING: there is no transaction in progress
2026-09-26 04:09:26,083 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 04:09:26,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:09:26,328 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 04:09:26,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 04:09:26,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 04:09:26,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 04:09:26,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xf24.c.gcp-upstream.internal','N') ;"
2026-09-26 04:09:27,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 04:09:27,357 [I] Table x_portal_user does not exist in database ranger
2026-09-26 04:09:27,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 04:09:27,602 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 04:09:27,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 04:09:27,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 04:09:28,093 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 04:09:28,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 04:09:29.157 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.175 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.193 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.209 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.226 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.243 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.312 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.322 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.340 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.353 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.365 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.381 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.392 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.400 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.408 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:29.416 UTC [12812] WARNING: there is no transaction in progress
2026-09-26 04:09:30,187 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 04:09:30,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 04:09:30,440 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 04:09:30,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 04:09:30,696 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 04:09:30,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 04:09:30,945 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 04:09:30,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 04:09:31,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 04:09:31,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xf24.c.gcp-upstream.internal';"
2026-09-26 04:09:31,712 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 04:09:31,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 04:09:31,959 [I] DB_PATCHES have already been applied
I20260926 04:09:31.965314 855 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrg1Sr7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 04:09:31.965389 855 mini_ranger.cc:193] Using host: 127.0.213.212
I20260926 04:09:31.968803 855 mini_ranger.cc:227] Ranger admin URL: http://127.0.213.212:56607
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:09:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 04:09:33.130330 855 mini_ranger.cc:160] Time spent starting Ranger: real 8.959s user 0.000s sys 0.009s
Sep 26, 2026 4:09:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:09:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 4:09:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:09:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:09:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 04:09:56.954115 855 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/rangerkms_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/rangerkms_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.213.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.213.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.213.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/HTTP_127.0.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 04:09:57 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395797, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 04:09:57.043335 855 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 04:09:57.052883 855 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 04:09:57.991245 855 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 04:09:57.991330 855 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms
I20260926 04:09:57.991353 855 mini_ranger_kms.cc:227] Using postgres at 127.0.213.212:50411
2026-09-26 04:09:58,158 [I] DB FLAVOR :POSTGRES
2026-09-26 04:09:58,159 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 04:09:58,159 [I] Checking connection
2026-09-26 04:09:58,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 04:09:58,412 [I] connection success
2026-09-26 04:09:58,412 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 04:09:58,413 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 04:09:58,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 04:09:58,682 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 04:09:58,682 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 04:09:58,683 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 04:09:58,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.213.212:50411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 04:09:58,999 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 04:09:59.533095 855 mini_ranger_kms.cc:310] Created kms service
I20260926 04:09:59.747820 855 mini_ranger_kms.cc:326] Created kudu user
I20260926 04:09:59.797184 855 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 04:09:59 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395797, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:00.224855 855 mini_ranger_kms.cc:384] Added ranger policy
I20260926 04:10:00.225370 855 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrg1Sr7/build/release/bin/postgresql.jar:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrg1Sr7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 04:10:00.225420 855 mini_ranger_kms.cc:240] Using host: 127.0.213.212
I20260926 04:10:00.231155 855 mini_ranger_kms.cc:276] Ranger KMS PID: 12993
I20260926 04:10:00.231251 855 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.213.212:59809
Sep 26, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:10:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:10:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 04:10:02.628686 855 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.585s user 0.006s sys 0.145s
I20260926 04:10:02.628798 855 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 04:10:02.628849 855 mini_ranger_kms.cc:401] 127.0.213.212:59809/kms/v1/keys
Sep 26 04:10:02 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395802, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.213.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 04:10:03 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395802, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.213.212@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
Sep 26 04:10:03 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395802, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.213.212@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
Sep 26 04:10:06 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:10:06.216404 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45103
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45103
--ranger_config_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:06.316615 13043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:06.316872 13043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:06.316897 13043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:06.318163 13043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:06.318243 13043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:06.318271 13043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:06.318287 13043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:06.318300 13043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:06.319988 13043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.213.212:59809/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45103
--ranger_config_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:06.320245 13043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:06.320429 13043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:06.326881 13049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:06.326958 13048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:06.326884 13051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:06.327267 13043 server_base.cc:1034] running on GCE node
I20260926 04:10:06.327582 13043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:06.327939 13043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:06.329134 13043 hybrid_clock.cc:648] HybridClock initialized: now 1790395806329104 us; error 76 us; skew 500 ppm
Sep 26 04:10:06 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:06.335623 13043 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:06.336248 13043 webserver.cc:460] Webserver started at http://127.0.213.254:41415/ using document root <none> and password file <none>
I20260926 04:10:06.336468 13043 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:06.336509 13043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:06.336611 13043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:10:06 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:06.398734 13043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "8637a4ae2ab246b696e246aff2a5af94"
format_stamp: "Formatted at 2026-09-26 04:10:06 on dist-test-slave-xf24"
server_key: "6255a9f565fe4678ba5a133996bb7308"
server_key_iv: "a3ba6cec0d8136a79d74b1b570153f75"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:06.399240 13043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "8637a4ae2ab246b696e246aff2a5af94"
format_stamp: "Formatted at 2026-09-26 04:10:06 on dist-test-slave-xf24"
server_key: "6255a9f565fe4678ba5a133996bb7308"
server_key_iv: "a3ba6cec0d8136a79d74b1b570153f75"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:06.401369 13043 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 04:10:06.418385 13058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:06.418674 13043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:06.418800 13043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "8637a4ae2ab246b696e246aff2a5af94"
format_stamp: "Formatted at 2026-09-26 04:10:06 on dist-test-slave-xf24"
server_key: "6255a9f565fe4678ba5a133996bb7308"
server_key_iv: "a3ba6cec0d8136a79d74b1b570153f75"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:06.418915 13043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:06.431455 13043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:06.433655 13043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:06.433827 13043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:06.438962 13043 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45103
I20260926 04:10:06.439031 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45103 every 8 connection(s)
I20260926 04:10:06.439400 13043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:06.440227 13111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:06.443674 13111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Bootstrap starting.
I20260926 04:10:06.444119 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13043
I20260926 04:10:06.444280 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:10:06.444482 855 external_mini_cluster.cc:1428] Setting key 487f83df4fd46c5290703913bc915922
I20260926 04:10:06.444499 13111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:06.444803 13111 log.cc:824] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:06.446117 13111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: No bootstrap required, opened a new log
I20260926 04:10:06.448020 13111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:06.448267 13111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:06.448338 13111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8637a4ae2ab246b696e246aff2a5af94, State: Initialized, Role: FOLLOWER
I20260926 04:10:06.448503 13111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:06.448607 13111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:06.448662 13111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:06.448720 13111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:06.449496 13111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:06.449609 13111 leader_election.cc:302] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8637a4ae2ab246b696e246aff2a5af94; no voters:
I20260926 04:10:06.449828 13111 leader_election.cc:288] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:10:06.449934 13116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:06.450110 13111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:06.450134 13116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 LEADER]: Becoming Leader. State: Replica: 8637a4ae2ab246b696e246aff2a5af94, State: Running, Role: LEADER
I20260926 04:10:06.450266 13116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:06.450609 13116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8637a4ae2ab246b696e246aff2a5af94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } } }
I20260926 04:10:06.450667 13116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:06.450845 13118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8637a4ae2ab246b696e246aff2a5af94. Latest consensus state: current_term: 1 leader_uuid: "8637a4ae2ab246b696e246aff2a5af94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } } }
I20260926 04:10:06.451064 13118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:06.451669 13120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:06.452211 13120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:06.452530 13111 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 04:10:06 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:06.457723 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:06.445257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.213.254:45103)
0926 04:10:06.445464 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:06.445475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:06.445514 (+ 39us) server_negotiation.cc:407] Connection header received
0926 04:10:06.446128 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:06.446144 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:06.446218 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:06.446395 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:06.447161 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:06.449002 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:06.450158 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:06.451016 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:06.451127 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:06.454247 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:06.454251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:06.454251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:06.454265 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:06.455942 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:06.456391 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:06.456407 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:06.456407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:06.456491 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:06.456709 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:06.456709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:06.456710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:06.456980 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:06.457165 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:06.457246 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:10:06.457365 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260926 04:10:06.468887 13120 catalog_manager.cc:1347] Generated new cluster ID: 97bab2738d844203b04f7bf527433fb6
I20260926 04:10:06.468984 13120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:06.478708 13120 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:10:06.479552 13120 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:06.488622 13120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Generated new TSK 0
I20260926 04:10:06.488965 13120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:10:07.192137 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45103
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:07.288518 13156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:07.288770 13156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:07.288812 13156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:07.290143 13156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:07.290210 13156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:07.290257 13156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:07.291841 13156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.213.212:59809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:07.292151 13156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:07.292438 13156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:07.297838 13161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:07.297837 13164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:07.298907 13162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:07.299202 13156 server_base.cc:1034] running on GCE node
I20260926 04:10:07.299419 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:07.299715 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:07.300984 13156 hybrid_clock.cc:648] HybridClock initialized: now 1790395807300969 us; error 32 us; skew 500 ppm
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:07.303845 13156 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:07.304418 13156 webserver.cc:460] Webserver started at http://127.0.213.193:34945/ using document root <none> and password file <none>
I20260926 04:10:07.304687 13156 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:07.304754 13156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:07.304900 13156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:07.316704 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "75d218f0c87c4ae39a44eaf8bb90a931"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "9c3b4cb88e217bb90bbeafcd9911f678"
server_key_iv: "842ace0c898d68d1daff2d0ffcb5d8c3"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.317227 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "75d218f0c87c4ae39a44eaf8bb90a931"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "9c3b4cb88e217bb90bbeafcd9911f678"
server_key_iv: "842ace0c898d68d1daff2d0ffcb5d8c3"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.324647 13156 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260926 04:10:07.341647 13171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.342154 13156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:07.342384 13156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "75d218f0c87c4ae39a44eaf8bb90a931"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "9c3b4cb88e217bb90bbeafcd9911f678"
server_key_iv: "842ace0c898d68d1daff2d0ffcb5d8c3"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.342554 13156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:07.373272 13156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:07.375836 13156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:07.376024 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:07.376403 13156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:07.376843 13156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:07.376904 13156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.376971 13156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:07.377004 13156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.384862 13156 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:43405
I20260926 04:10:07.385265 13156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:10:07.385895 13286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:43405 every 8 connection(s)
I20260926 04:10:07.391366 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13156
I20260926 04:10:07.391486 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:10:07.391703 855 external_mini_cluster.cc:1428] Setting key b6116692a40b5193219485e7b33bdc52
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:07.396701 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:07.386042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45103 (local address 127.0.213.193:38103)
0926 04:10:07.386210 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:07.386256 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:07.386994 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:07.387275 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:07.387285 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:07.387483 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:07.387684 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:07.387689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.388537 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.388539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:07.389491 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:07.389494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.389995 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.389996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:07.390125 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:07.390737 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:07.390747 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:07.392398 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:07.394726 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:07.394735 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:07.394738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:07.395116 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:07.395364 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:07.395365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:07.395365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:07.395519 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:07.395857 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:07.395864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:07.396095 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 04:10:07.396210 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 04:10:07.396392 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260926 04:10:07.397264 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:07.386373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:38103 (local address 127.0.213.254:45103)
0926 04:10:07.386543 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:07.386545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:07.386556 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:10:07.387119 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:07.387120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:07.387136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:07.387203 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:07.387786 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.388422 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.389638 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.389874 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.389956 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:07.392512 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:07.392514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:07.392514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:07.392522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:07.394614 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:07.395212 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:07.395213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:07.395213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:07.395283 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:07.395594 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:07.395595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:07.395595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:07.395749 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:07.395830 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:07.397044 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0926 04:10:07.397163 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260926 04:10:07.397934 13287 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45103
I20260926 04:10:07.398074 13287 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:07.398284 13287 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:07.398983 13075 ts_manager.cc:194] Registered new tserver with Master: 75d218f0c87c4ae39a44eaf8bb90a931 (127.0.213.193:43405)
I20260926 04:10:07.400156 13075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:38103
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:10:07.431784 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45103
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 04:10:07.561125 13293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:07.561429 13293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:07.561466 13293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:07.563433 13293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:07.563519 13293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:07.563566 13293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:07.565936 13293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.213.212:59809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:07.566270 13293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:07.566577 13293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:07.572932 13303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:07.572971 13301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:07.573153 13300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:07.574175 13293 server_base.cc:1034] running on GCE node
I20260926 04:10:07.574406 13293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:07.574694 13293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:07.575855 13293 hybrid_clock.cc:648] HybridClock initialized: now 1790395807575832 us; error 44 us; skew 500 ppm
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:07.579198 13293 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:07.579758 13293 webserver.cc:460] Webserver started at http://127.0.213.194:40837/ using document root <none> and password file <none>
I20260926 04:10:07.580091 13293 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:07.580152 13293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:07.580287 13293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:07.595487 13293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "4573a72db6bb49698f903516cfa0730c"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "807d195cbb033304ac7a357e5ab37f06"
server_key_iv: "e4c545cc16b944530e47a7170ec05cdc"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.595927 13293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "4573a72db6bb49698f903516cfa0730c"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "807d195cbb033304ac7a357e5ab37f06"
server_key_iv: "e4c545cc16b944530e47a7170ec05cdc"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.604575 13293 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.001s
I20260926 04:10:07.623874 13310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.624202 13293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 04:10:07.624308 13293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "4573a72db6bb49698f903516cfa0730c"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "807d195cbb033304ac7a357e5ab37f06"
server_key_iv: "e4c545cc16b944530e47a7170ec05cdc"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.624418 13293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:07.647600 13293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:07.650322 13293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:07.650545 13293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:07.650856 13293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:07.651321 13293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:07.651422 13293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.651507 13293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:07.651572 13293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.658370 13293 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:33291
I20260926 04:10:07.658955 13293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:10:07.661998 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13293
I20260926 04:10:07.662107 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:10:07.662315 855 external_mini_cluster.cc:1428] Setting key aa5733769129192e86501f547099552c
I20260926 04:10:07.666740 13424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:33291 every 8 connection(s)
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:10:07.679848 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:07.667880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:55679 (local address 127.0.213.254:45103)
0926 04:10:07.667951 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:07.667954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:07.668323 (+ 369us) server_negotiation.cc:407] Connection header received
0926 04:10:07.669347 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:07.669348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:07.669368 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:07.669453 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:07.670319 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.671075 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.672654 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.672988 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.673039 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:07.675955 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:07.675958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:07.675959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:07.675968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:07.677918 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:07.678572 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:07.678573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:07.678574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:07.678641 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:07.678995 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:07.678996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:07.678996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:07.679146 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:07.679192 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:07.679635 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 04:10:07.679736 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:10:07.680037 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:07.667543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45103 (local address 127.0.213.194:55679)
0926 04:10:07.668172 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:07.668229 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:07.669200 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:07.669572 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:07.669577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:07.669853 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:07.670183 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:07.670192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.671202 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.671203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:07.672533 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:07.672537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.673161 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.673162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:07.673312 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:07.673992 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:07.674004 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:07.675816 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:07.678057 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:07.678066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:07.678069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:07.678430 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:07.678726 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:07.678727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:07.678727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:07.678883 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:07.679243 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:07.679250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:07.679440 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 04:10:07.679583 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 04:10:07.679762 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":100,"threads_started":1}
I20260926 04:10:07.680821 13425 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45103
I20260926 04:10:07.680948 13425 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:07.681145 13425 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:07.681604 13075 ts_manager.cc:194] Registered new tserver with Master: 4573a72db6bb49698f903516cfa0730c (127.0.213.194:33291)
I20260926 04:10:07.682107 13075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:55679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:10:07.707826 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45103
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:07.847733 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:07.848176 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:07.848290 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:07.850355 13430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:07.850533 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:07.850630 13430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:07.853261 13430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.213.212:59809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:07.853817 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:07.854182 13430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:07.861496 13436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:07.861585 13438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:07.861997 13430 server_base.cc:1034] running on GCE node
W20260926 04:10:07.861474 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:07.862340 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:07.862620 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:07.863801 13430 hybrid_clock.cc:648] HybridClock initialized: now 1790395807863778 us; error 40 us; skew 500 ppm
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:07.867120 13430 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:07.867702 13430 webserver.cc:460] Webserver started at http://127.0.213.195:37867/ using document root <none> and password file <none>
I20260926 04:10:07.867990 13430 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:07.868083 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:07.868239 13430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:07.881939 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "aa2c702320d15988dc77b16a80a9bec3"
server_key_iv: "8d7a3d4e4e08eb20fc485385f291509b"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.882462 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "aa2c702320d15988dc77b16a80a9bec3"
server_key_iv: "8d7a3d4e4e08eb20fc485385f291509b"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.884508 13430 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260926 04:10:07.909572 13445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.909765 13430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:07.909854 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "aa2c702320d15988dc77b16a80a9bec3"
server_key_iv: "8d7a3d4e4e08eb20fc485385f291509b"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:07.909962 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:07.931699 13430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:07.934553 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:07.934737 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:07.934966 13430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:07.935406 13430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:07.935451 13430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.935485 13430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:07.935505 13430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:07.949579 13430 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:35423
I20260926 04:10:07.950024 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:10:07.951422 13558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:35423 every 8 connection(s)
I20260926 04:10:07.956223 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13430
I20260926 04:10:07.956344 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:10:07.956522 855 external_mini_cluster.cc:1428] Setting key 80065a090afb73a2f65d9b40aa8394e9
Sep 26 04:10:07 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:07.965106 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:07.952160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:38699 (local address 127.0.213.254:45103)
0926 04:10:07.952218 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:07.952221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:07.952947 (+ 726us) server_negotiation.cc:407] Connection header received
0926 04:10:07.953651 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:07.953652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:07.953665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:07.953696 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:07.954536 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.955288 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.958406 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.958724 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.958773 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:07.961623 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:07.961624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:07.961624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:07.961633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:07.963340 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:07.963886 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:07.963887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:07.963888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:07.963966 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:07.964324 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:07.964324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:07.964325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:07.964476 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:07.964529 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:07.964903 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 04:10:07.965008 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:10:07.965962 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:07.952072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45103 (local address 127.0.213.195:38699)
0926 04:10:07.952806 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:07.952855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:07.953524 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:07.953881 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:07.953885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:07.954145 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:07.954425 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:07.954431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.955410 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.955410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:07.958292 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:07.958296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:07.958892 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:07.958893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:07.959061 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:07.959726 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:07.959738 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:07.961499 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:07.963445 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:07.963455 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:07.963458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:07.963791 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:07.964034 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:07.964035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:07.964035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:07.964240 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:07.964564 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:07.964571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:07.964750 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 04:10:07.965520 (+ 770us) client_negotiation.cc:240] Negotiation successful
0926 04:10:07.965708 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":86,"threads_started":1}
I20260926 04:10:07.966552 13559 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45103
I20260926 04:10:07.966660 13559 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:07.966859 13559 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:07.967278 13075 ts_manager.cc:194] Registered new tserver with Master: f84ff8e80f3e4f44ac1f07e66c54a7b3 (127.0.213.195:35423)
I20260926 04:10:07.967828 13075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:38699
I20260926 04:10:07.969094 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:10:07.969935 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13156
I20260926 04:10:07.976747 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13293
I20260926 04:10:07.984210 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13430
I20260926 04:10:07.989550 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13043
I20260926 04:10:08.002602 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45103
--webserver_interface=127.0.213.254
--webserver_port=41415
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45103
--ranger_config_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:08.086694 13562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:08.086961 13562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:08.087002 13562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:08.088316 13562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:08.088382 13562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:08.088406 13562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:08.088435 13562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:08.088490 13562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:08.090169 13562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45103
--ranger_config_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:08.090485 13562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:08.090732 13562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:08.095939 13570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.095937 13567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.096273 13568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:08.097420 13562 server_base.cc:1034] running on GCE node
I20260926 04:10:08.097726 13562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:08.098189 13562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:08.099388 13562 hybrid_clock.cc:648] HybridClock initialized: now 1790395808099348 us; error 56 us; skew 500 ppm
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:08.102972 13562 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:08.103600 13562 webserver.cc:460] Webserver started at http://127.0.213.254:41415/ using document root <none> and password file <none>
I20260926 04:10:08.103935 13562 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:08.104031 13562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:08.173529 13562 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:08.174293 13578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.174499 13562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:08.174592 13562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "8637a4ae2ab246b696e246aff2a5af94"
format_stamp: "Formatted at 2026-09-26 04:10:06 on dist-test-slave-xf24"
server_key: "6255a9f565fe4678ba5a133996bb7308"
server_key_iv: "a3ba6cec0d8136a79d74b1b570153f75"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:08.174975 13562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:08.184254 13562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:08.186595 13562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:08.186762 13562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:08.192024 13562 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45103
I20260926 04:10:08.192078 13630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45103 every 8 connection(s)
I20260926 04:10:08.192498 13562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:08.194242 13631 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:10:08.195191 13631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Bootstrap starting.
I20260926 04:10:08.200227 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13562
I20260926 04:10:08.200551 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:43405
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:45103
--webserver_port=34945
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:08.205490 13631 log.cc:824] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:08.207885 13631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:08.208285 13631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Bootstrap complete.
I20260926 04:10:08.211699 13631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:08.212087 13631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8637a4ae2ab246b696e246aff2a5af94, State: Initialized, Role: FOLLOWER
I20260926 04:10:08.212257 13631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:08.212374 13631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:08.212453 13631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:08.212509 13631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:10:08.213999 13631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:08.214118 13631 leader_election.cc:302] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8637a4ae2ab246b696e246aff2a5af94; no voters:
I20260926 04:10:08.214342 13631 leader_election.cc:288] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:10:08.214411 13635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:10:08.214641 13635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [term 2 LEADER]: Becoming Leader. State: Replica: 8637a4ae2ab246b696e246aff2a5af94, State: Running, Role: LEADER
I20260926 04:10:08.214699 13631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:08.214828 13635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } }
I20260926 04:10:08.215098 13635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8637a4ae2ab246b696e246aff2a5af94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } } }
I20260926 04:10:08.215163 13635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:08.215293 13635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8637a4ae2ab246b696e246aff2a5af94. Latest consensus state: current_term: 2 leader_uuid: "8637a4ae2ab246b696e246aff2a5af94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8637a4ae2ab246b696e246aff2a5af94" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45103 } } }
I20260926 04:10:08.215348 13635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94 [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:08.215824 13641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:08.216275 13631 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 04:10:08.216572 13641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:08.219694 13641 catalog_manager.cc:1259] Loaded cluster ID: 97bab2738d844203b04f7bf527433fb6
I20260926 04:10:08.219815 13641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:08.221742 13641 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:08.222084 13641 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Loaded TSK: 0
I20260926 04:10:08.222398 13641 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:10:08.319224 13633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:08.319481 13633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:08.319527 13633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:08.321235 13633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:08.321544 13633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:08.321637 13633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:08.323767 13633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:43405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=34945
--tserver_master_addrs=127.0.213.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:08.324133 13633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:08.324496 13633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:08.330394 13671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.330535 13668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:08.331277 13633 server_base.cc:1034] running on GCE node
W20260926 04:10:08.332190 13669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:08.332700 13633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:08.333118 13633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:08.334302 13633 hybrid_clock.cc:648] HybridClock initialized: now 1790395808334283 us; error 37 us; skew 500 ppm
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:08.337661 13633 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:08.340094 13633 webserver.cc:460] Webserver started at http://127.0.213.193:34945/ using document root <none> and password file <none>
I20260926 04:10:08.340426 13633 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:08.340535 13633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:08.358456 13633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260926 04:10:08.360247 13679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.360472 13633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:08.360755 13633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "75d218f0c87c4ae39a44eaf8bb90a931"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "9c3b4cb88e217bb90bbeafcd9911f678"
server_key_iv: "842ace0c898d68d1daff2d0ffcb5d8c3"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:08.361181 13633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:08.377804 13633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:08.380472 13633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:08.380719 13633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:08.381322 13633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:08.382079 13633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:08.382333 13633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.382503 13633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:08.382612 13633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.391134 13633 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:43405
I20260926 04:10:08.391386 13792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:43405 every 8 connection(s)
I20260926 04:10:08.391700 13633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:10:08.398857 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13633
I20260926 04:10:08.399169 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:33291
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:45103
--webserver_port=40837
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:08.408596 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.392556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45103 (local address 127.0.213.193:56935)
0926 04:10:08.393073 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:08.393108 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:08.393709 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:08.396818 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:08.396823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:08.397113 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:08.397404 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:08.397411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.399218 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.399220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:08.400612 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:08.400615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.401521 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.401522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:08.401661 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.402347 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:08.402358 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:08.403857 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:08.406695 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:08.406702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:08.406704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:08.407048 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:08.407302 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:08.407303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:08.407303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:08.407432 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:08.407909 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:08.407915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:08.408088 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 04:10:08.408185 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 04:10:08.408352 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":92,"threads_started":1}
I20260926 04:10:08.409699 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.392628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:56935 (local address 127.0.213.254:45103)
0926 04:10:08.396243 (+ 3615us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:08.396254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:08.396301 (+ 47us) server_negotiation.cc:407] Connection header received
0926 04:10:08.396511 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:08.396521 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:08.396586 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:08.397564 (+ 978us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:08.397668 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.399083 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.400741 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.401380 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.401984 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.403987 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:08.403993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:08.403993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:08.404007 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:08.406576 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.407132 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.407135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.407135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.407217 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.407510 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.407511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.407511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.407816 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:08.409237 (+ 1421us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:08.409329 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 04:10:08.409454 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3547,"thread_start_us":84,"threads_started":1}
I20260926 04:10:08.411162 13793 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45103
I20260926 04:10:08.411276 13793 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.411468 13793 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
W20260926 04:10:08.412407 13793 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45103 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 04:10:08.412575 13793 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.412698 13793 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:08.413056 13793 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.413182 13793 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
W20260926 04:10:08.413537 13793 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:10:08.520972 13797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:08.521288 13797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:08.521327 13797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:08.523183 13797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:08.523252 13797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:08.523298 13797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:08.525348 13797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:33291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=40837
--tserver_master_addrs=127.0.213.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:08.525641 13797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:08.525920 13797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:08.532414 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.532414 13803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.532925 13805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:08.533735 13797 server_base.cc:1034] running on GCE node
I20260926 04:10:08.533991 13797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:08.534298 13797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:08.535482 13797 hybrid_clock.cc:648] HybridClock initialized: now 1790395808535449 us; error 47 us; skew 500 ppm
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:08.540350 13797 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:08.540928 13797 webserver.cc:460] Webserver started at http://127.0.213.194:40837/ using document root <none> and password file <none>
I20260926 04:10:08.541249 13797 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:08.541373 13797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:08.564565 13797 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:10:08.565526 13813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.565812 13797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:08.565917 13797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "4573a72db6bb49698f903516cfa0730c"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "807d195cbb033304ac7a357e5ab37f06"
server_key_iv: "e4c545cc16b944530e47a7170ec05cdc"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:08.566394 13797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:08.583768 13797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:08.586218 13797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:08.586436 13797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:08.586727 13797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:08.587105 13797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:08.587159 13797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.587204 13797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:08.587232 13797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.594707 13797 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:33291
I20260926 04:10:08.595144 13797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:10:08.595800 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:33291 every 8 connection(s)
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:08.604230 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13797
I20260926 04:10:08.604553 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.213.212:59809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:35423
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:45103
--webserver_port=37867
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:08.605639 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.596233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:47901 (local address 127.0.213.254:45103)
0926 04:10:08.596294 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:08.596297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:08.596461 (+ 164us) server_negotiation.cc:407] Connection header received
0926 04:10:08.597019 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:08.597020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:08.597036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:08.597084 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:08.597685 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.598192 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.599230 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.599445 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.599522 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.602105 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:08.602108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:08.602108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:08.602117 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:08.603905 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.604509 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.604510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.604511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.604573 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.604864 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.604865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.604865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.605007 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:08.605060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:08.605426 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 04:10:08.605536 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:10:08.606212 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.595983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45103 (local address 127.0.213.194:47901)
0926 04:10:08.596334 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:08.596370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:08.596890 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:08.597163 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:08.597167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:08.597371 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:08.597573 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:08.597577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.598288 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.598289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:08.599132 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:08.599134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.599556 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.599557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:08.599683 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.600336 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:08.600345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:08.601978 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:08.604018 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:08.604028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:08.604031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:08.604401 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:08.604648 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:08.604649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:08.604649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:08.604788 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:08.605092 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:08.605097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:08.605272 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 04:10:08.605776 (+ 504us) client_negotiation.cc:240] Negotiation successful
0926 04:10:08.605957 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260926 04:10:08.606835 13927 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45103
I20260926 04:10:08.606937 13927 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.607155 13927 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
W20260926 04:10:08.608150 13927 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45103 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 04:10:08.608227 13927 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.608294 13927 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:08.608667 13927 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.608740 13927 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
W20260926 04:10:08.609102 13927 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:10:08.723838 13930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:08.724254 13930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:08.724355 13930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:08.726291 13930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:08.726418 13930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:08.726505 13930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:08.728900 13930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.213.212:59809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:35423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=37867
--tserver_master_addrs=127.0.213.254:45103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:08.729230 13930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:08.729506 13930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:08.735253 13938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.735512 13936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:08.735929 13935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:08.736194 13930 server_base.cc:1034] running on GCE node
I20260926 04:10:08.736599 13930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:08.737064 13930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:08.738265 13930 hybrid_clock.cc:648] HybridClock initialized: now 1790395808738229 us; error 52 us; skew 500 ppm
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:08.742172 13930 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:08.742746 13930 webserver.cc:460] Webserver started at http://127.0.213.195:37867/ using document root <none> and password file <none>
I20260926 04:10:08.743010 13930 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:08.743098 13930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:08.777760 13930 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.001s sys 0.000s
I20260926 04:10:08.784246 13947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.784534 13930 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 04:10:08.784643 13930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3"
format_stamp: "Formatted at 2026-09-26 04:10:07 on dist-test-slave-xf24"
server_key: "aa2c702320d15988dc77b16a80a9bec3"
server_key_iv: "8d7a3d4e4e08eb20fc485385f291509b"
server_key_version: "kuduclusterkey@0"
I20260926 04:10:08.785163 13930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:08.797467 13930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:08.800462 13930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:08.800628 13930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:08.800938 13930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:08.801285 13930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:08.801340 13930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.801380 13930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:08.801412 13930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:08.808595 13930 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:35423
I20260926 04:10:08.808704 14060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:35423 every 8 connection(s)
I20260926 04:10:08.809126 13930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:10:08.812691 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 13930
Sep 26 04:10:08 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:08.823194 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.810268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:56663 (local address 127.0.213.254:45103)
0926 04:10:08.810316 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:08.810318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:08.810329 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:10:08.811085 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:08.811088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:08.811101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:08.811172 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:08.812007 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.812829 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.814462 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.814727 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.814810 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.818315 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:08.818316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:08.818316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:08.818323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:08.819976 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.821181 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.821182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.821183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.821255 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.821878 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.821880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.821880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.822048 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:08.822107 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:08.822978 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 04:10:08.823083 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:10:08.823431 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.809954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45103 (local address 127.0.213.195:56663)
0926 04:10:08.810160 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:08.810210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:08.810841 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:08.811252 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:08.811257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:08.811580 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:08.811872 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:08.811879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.812967 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.812968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:08.814357 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:08.814361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.814829 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.814831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:08.814968 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.816561 (+ 1593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:08.816572 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:08.818198 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:08.820119 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:08.820131 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:08.820134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:08.821068 (+ 934us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:08.821604 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:08.821604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:08.821605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:08.821782 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:08.822161 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:08.822169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:08.822773 (+ 604us) client_negotiation.cc:769] Sending connection context
0926 04:10:08.822949 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 04:10:08.823165 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260926 04:10:08.823884 14061 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45103
I20260926 04:10:08.824007 14061 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.824154 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:08.813391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.0.213.254:45103)
0926 04:10:08.814397 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:08.814398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:08.814407 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:10:08.814458 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:08.814459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:08.814472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:08.814550 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:08.815406 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.816176 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.817276 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:08.817604 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:08.817689 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:08.820121 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:08.820122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:08.820122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:08.820130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:08.822235 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.822657 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.822659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.822659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.822731 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:08.823371 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:08.823372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:08.823373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:08.823510 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:08.823816 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:08.823906 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 04:10:08.824000 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260926 04:10:08.824249 14061 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
W20260926 04:10:08.824899 14061 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45103 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 04:10:08.824999 14061 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.825094 14061 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:08.825423 14061 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:08.825533 14061 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
W20260926 04:10:08.825867 14061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:10:09.413785 13793 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:09.413960 13793 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:09.414433 13595 ts_manager.cc:194] Registered new tserver with Master: 75d218f0c87c4ae39a44eaf8bb90a931 (127.0.213.193:43405)
I20260926 04:10:09.415441 13595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:56935
Sep 26 04:10:09 dist-test-slave-xf24 krb5kdc[12599](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:09.609308 13927 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:09.609491 13927 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:09.609951 13595 ts_manager.cc:194] Registered new tserver with Master: 4573a72db6bb49698f903516cfa0730c (127.0.213.194:33291)
I20260926 04:10:09.610502 13595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:47901
I20260926 04:10:09.826087 14061 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:09.826267 14061 heartbeater.cc:511] Master 127.0.213.254:45103 requested a full tablet report, sending...
I20260926 04:10:09.827081 13595 ts_manager.cc:194] Registered new tserver with Master: f84ff8e80f3e4f44ac1f07e66c54a7b3 (127.0.213.195:35423)
I20260926 04:10:09.827639 13595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:56663
I20260926 04:10:09.829414 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:10:09.839699 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.830332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.0.213.254:45103)
0926 04:10:09.830599 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.830601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.830764 (+ 163us) server_negotiation.cc:407] Connection header received
0926 04:10:09.830789 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.830789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.830803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.830861 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:09.831617 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.832429 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.833476 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.833806 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.833886 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.836189 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:09.836191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:09.836191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:09.836199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:09.838077 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:09.838518 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:09.838519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:09.838520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:09.838592 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:09.838820 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:09.838821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:09.838822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:09.838964 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:09.839065 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.839483 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.839588 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":61,"threads_started":1}
I20260926 04:10:09.846858 13595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:10:09.849210 13595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 04:10:09 dist-test-slave-xf24 krb5kdc[12599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for HTTP/127.0.213.212@KRBTEST.COM
I20260926 04:10:09.870357 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.853364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:35423 (local address 127.0.0.1:47704)
0926 04:10:09.858916 (+ 5552us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:09.858924 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:09.858966 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:09.859603 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:09.859606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:09.859608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:09.859692 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.859694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.864826 (+ 5132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.864827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.868936 (+ 4109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.868939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.870001 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.870002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.870121 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.870139 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 04:10:09.870192 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:10:09.870255 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5538,"thread_start_us":67,"threads_started":1}
I20260926 04:10:09.871456 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.853456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.0.213.195:35423)
0926 04:10:09.858867 (+ 5411us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.858872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.859014 (+ 142us) server_negotiation.cc:407] Connection header received
0926 04:10:09.859054 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.859057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.859074 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.859149 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:09.859777 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.864734 (+ 4957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.869035 (+ 4301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.869867 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.871185 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.871292 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.871323 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.871375 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5374,"thread_start_us":54,"threads_started":1}
I20260926 04:10:09.871856 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.853787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.0.213.194:33291)
0926 04:10:09.858492 (+ 4705us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.858499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.859370 (+ 871us) server_negotiation.cc:407] Connection header received
0926 04:10:09.859411 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.859413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.859429 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.859494 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:09.861336 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.862475 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.866282 (+ 3807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.867136 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.867213 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.867284 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.871714 (+ 4430us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.871777 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4658,"thread_start_us":53,"threads_started":1}
I20260926 04:10:09.870932 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.853543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:33291 (local address 127.0.0.1:56646)
0926 04:10:09.859256 (+ 5713us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:09.859266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:09.859310 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:09.861201 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:09.861202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:09.861203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:09.861249 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.861251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.862582 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.862583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.863982 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.863986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.870685 (+ 6699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.870686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.870752 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.870757 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:09.870812 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:10:09.870858 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5697,"thread_start_us":58,"threads_started":1}
I20260926 04:10:09.875370 13861 tablet_service.cc:1467] Processing CreateTablet for tablet abdedad8ee1a44f2ab3864d80161136c (DEFAULT_TABLE table=test-table [id=a428ac811a7441dabe76aef35dcb8569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:09.875504 13995 tablet_service.cc:1467] Processing CreateTablet for tablet abdedad8ee1a44f2ab3864d80161136c (DEFAULT_TABLE table=test-table [id=a428ac811a7441dabe76aef35dcb8569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:09.875835 13861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdedad8ee1a44f2ab3864d80161136c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:09.875862 13995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdedad8ee1a44f2ab3864d80161136c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:09.876405 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.853134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43405 (local address 127.0.0.1:36664)
0926 04:10:09.858589 (+ 5455us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:09.858651 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:09.858721 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:09.867557 (+ 8836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:09.867559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:09.867562 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:09.867611 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.867613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.872577 (+ 4964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.872579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.874163 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.874166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.876113 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.876114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.876187 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.876192 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:09.876255 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:10:09.876311 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5417,"thread_start_us":77,"threads_started":1}
I20260926 04:10:09.876904 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.853220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.0.213.193:43405)
0926 04:10:09.866180 (+ 12960us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.866187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.866199 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:10:09.867330 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.867334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.867358 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.867439 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:09.867691 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.872462 (+ 4771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.875143 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.875975 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.876084 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.876155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.876751 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.876807 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12912,"thread_start_us":42,"threads_started":1}
I20260926 04:10:09.879624 14095 tablet_bootstrap.cc:492] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Bootstrap starting.
I20260926 04:10:09.880506 14095 tablet_bootstrap.cc:654] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:09.880615 13727 tablet_service.cc:1467] Processing CreateTablet for tablet abdedad8ee1a44f2ab3864d80161136c (DEFAULT_TABLE table=test-table [id=a428ac811a7441dabe76aef35dcb8569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:09.880914 14095 log.cc:824] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:09.880968 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdedad8ee1a44f2ab3864d80161136c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:09.885521 14098 tablet_bootstrap.cc:492] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931: Bootstrap starting.
I20260926 04:10:09.886418 14098 tablet_bootstrap.cc:654] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:09.886813 14098 log.cc:824] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:09.888300 14096 tablet_bootstrap.cc:492] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Bootstrap starting.
I20260926 04:10:09.889323 14096 tablet_bootstrap.cc:654] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:09.889721 14096 log.cc:824] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:09.890846 14095 tablet_bootstrap.cc:492] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: No bootstrap required, opened a new log
I20260926 04:10:09.891010 14095 ts_tablet_manager.cc:1397] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260926 04:10:09.894922 14095 raft_consensus.cc:348] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.895699 14095 raft_consensus.cc:374] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:09.895824 14095 raft_consensus.cc:729] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4573a72db6bb49698f903516cfa0730c, State: Initialized, Role: FOLLOWER
I20260926 04:10:09.896214 14095 consensus_queue.cc:260] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.896760 14095 ts_tablet_manager.cc:1428] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260926 04:10:09.897015 13927 heartbeater.cc:503] Master 127.0.213.254:45103 was elected leader, sending a full tablet report...
I20260926 04:10:09.899474 14096 tablet_bootstrap.cc:492] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: No bootstrap required, opened a new log
I20260926 04:10:09.899637 14096 ts_tablet_manager.cc:1397] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 04:10:09.903095 14096 raft_consensus.cc:348] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.903415 14096 raft_consensus.cc:374] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:09.903496 14096 raft_consensus.cc:729] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84ff8e80f3e4f44ac1f07e66c54a7b3, State: Initialized, Role: FOLLOWER
I20260926 04:10:09.903726 14096 consensus_queue.cc:260] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.904135 14096 ts_tablet_manager.cc:1428] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 04:10:09.904310 14061 heartbeater.cc:503] Master 127.0.213.254:45103 was elected leader, sending a full tablet report...
I20260926 04:10:09.906565 14098 tablet_bootstrap.cc:492] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931: No bootstrap required, opened a new log
I20260926 04:10:09.906656 14098 ts_tablet_manager.cc:1397] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20260926 04:10:09.909350 14098 raft_consensus.cc:348] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.909696 14098 raft_consensus.cc:374] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:09.909754 14098 raft_consensus.cc:729] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d218f0c87c4ae39a44eaf8bb90a931, State: Initialized, Role: FOLLOWER
I20260926 04:10:09.910084 14098 consensus_queue.cc:260] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.910508 14098 ts_tablet_manager.cc:1428] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 04:10:09.910697 13793 heartbeater.cc:503] Master 127.0.213.254:45103 was elected leader, sending a full tablet report...
I20260926 04:10:09.929349 14102 raft_consensus.cc:482] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:09.929495 14102 raft_consensus.cc:504] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.929942 14102 leader_election.cc:288] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4573a72db6bb49698f903516cfa0730c (127.0.213.194:33291), 75d218f0c87c4ae39a44eaf8bb90a931 (127.0.213.193:43405)
I20260926 04:10:09.934926 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.930598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:47263 (local address 127.0.213.193:43405)
0926 04:10:09.930643 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.930646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.931172 (+ 526us) server_negotiation.cc:407] Connection header received
0926 04:10:09.931283 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.931285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.931294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.931325 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:09.931583 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.932368 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.933869 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.934473 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.934530 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.934540 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.934778 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.934825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:10:09.935451 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.930283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43405 (local address 127.0.213.195:47263)
0926 04:10:09.931090 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:09.931102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:09.931204 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:09.931399 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:09.931400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:09.931402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:09.931497 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.931500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.932479 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.932479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.933753 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.933757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.934608 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.934609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.934691 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.934695 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:10:09.935286 (+ 591us) client_negotiation.cc:240] Negotiation successful
0926 04:10:09.935345 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":100,"threads_started":1}
I20260926 04:10:09.936087 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdedad8ee1a44f2ab3864d80161136c" candidate_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" is_pre_election: true
I20260926 04:10:09.936280 13729 raft_consensus.cc:2454] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84ff8e80f3e4f44ac1f07e66c54a7b3 in term 0.
I20260926 04:10:09.936571 13948 leader_election.cc:302] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75d218f0c87c4ae39a44eaf8bb90a931, f84ff8e80f3e4f44ac1f07e66c54a7b3; no voters:
I20260926 04:10:09.936717 14102 raft_consensus.cc:2781] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:10:09.936761 14102 raft_consensus.cc:482] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:10:09.936888 14102 raft_consensus.cc:3037] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:09.938068 14102 raft_consensus.cc:504] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.938222 14102 leader_election.cc:288] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [CANDIDATE]: Term 1 election: Requested vote from peers 4573a72db6bb49698f903516cfa0730c (127.0.213.194:33291), 75d218f0c87c4ae39a44eaf8bb90a931 (127.0.213.193:43405)
I20260926 04:10:09.938721 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdedad8ee1a44f2ab3864d80161136c" candidate_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75d218f0c87c4ae39a44eaf8bb90a931"
I20260926 04:10:09.938820 13729 raft_consensus.cc:3037] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:09.940307 13729 raft_consensus.cc:2454] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84ff8e80f3e4f44ac1f07e66c54a7b3 in term 1.
I20260926 04:10:09.940899 13948 leader_election.cc:302] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75d218f0c87c4ae39a44eaf8bb90a931, f84ff8e80f3e4f44ac1f07e66c54a7b3; no voters:
I20260926 04:10:09.941038 14102 raft_consensus.cc:2781] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:09.941192 14102 raft_consensus.cc:686] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 1 LEADER]: Becoming Leader. State: Replica: f84ff8e80f3e4f44ac1f07e66c54a7b3, State: Running, Role: LEADER
I20260926 04:10:09.941298 14102 consensus_queue.cc:237] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } }
I20260926 04:10:09.942257 13595 catalog_manager.cc:5495] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f84ff8e80f3e4f44ac1f07e66c54a7b3 (127.0.213.195). New cstate: current_term: 1 leader_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 35423 } health_report { overall_health: HEALTHY } } }
I20260926 04:10:09.947604 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.930985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:46785 (local address 127.0.213.194:33291)
0926 04:10:09.931028 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.931031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.943842 (+ 12811us) server_negotiation.cc:407] Connection header received
0926 04:10:09.944244 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.944245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.944256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.944286 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:09.944491 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.945229 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.946568 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.947191 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.947246 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.947256 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.947466 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.947519 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:10:09.948132 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.930775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:33291 (local address 127.0.213.195:46785)
0926 04:10:09.943559 (+ 12784us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:09.943574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:09.944141 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:09.944353 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:09.944354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:09.944355 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:09.944418 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.944420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.945343 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.945344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.946447 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:09.946451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.947319 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.947320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:09.947385 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.947388 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:10:09.947955 (+ 567us) client_negotiation.cc:240] Negotiation successful
0926 04:10:09.948013 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12767,"thread_start_us":91,"threads_started":1}
I20260926 04:10:09.948679 13881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdedad8ee1a44f2ab3864d80161136c" candidate_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4573a72db6bb49698f903516cfa0730c" is_pre_election: true
I20260926 04:10:09.948845 13881 raft_consensus.cc:2454] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84ff8e80f3e4f44ac1f07e66c54a7b3 in term 0.
I20260926 04:10:09.948977 13880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdedad8ee1a44f2ab3864d80161136c" candidate_uuid: "f84ff8e80f3e4f44ac1f07e66c54a7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4573a72db6bb49698f903516cfa0730c"
I20260926 04:10:09.949039 13880 raft_consensus.cc:3037] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:09.950160 13880 raft_consensus.cc:2454] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84ff8e80f3e4f44ac1f07e66c54a7b3 in term 1.
I20260926 04:10:09.961118 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:09.957095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.0.213.195:35423)
0926 04:10:09.957184 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:09.957186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:09.957202 (+ 16us) server_negotiation.cc:407] Connection header received
0926 04:10:09.957254 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:09.957255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:09.957281 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:09.957392 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:09.957596 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.958392 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.959573 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:09.959979 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:09.960154 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:09.960390 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:09.960580 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:10:09.960677 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:09.960983 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 04:10:09.961012 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:10:09.963569 13729 raft_consensus.cc:1261] T abdedad8ee1a44f2ab3864d80161136c P 75d218f0c87c4ae39a44eaf8bb90a931 [term 1 FOLLOWER]: Refusing update from remote peer f84ff8e80f3e4f44ac1f07e66c54a7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:09.964075 14106 consensus_queue.cc:1035] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75d218f0c87c4ae39a44eaf8bb90a931" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:09.966722 13880 raft_consensus.cc:1261] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 1 FOLLOWER]: Refusing update from remote peer f84ff8e80f3e4f44ac1f07e66c54a7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:09.967135 14106 consensus_queue.cc:1035] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4573a72db6bb49698f903516cfa0730c" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:09.974215 14109 mvcc.cc:205] Tried to move back new op lower bound from 7333461237607428096 to 7333461237520289792. Current Snapshot: MvccSnapshot[applied={T|T < 7333461237607428096}]
I20260926 04:10:09.974464 14112 mvcc.cc:205] Tried to move back new op lower bound from 7333461237607428096 to 7333461237520289792. Current Snapshot: MvccSnapshot[applied={T|T < 7333461237607428096}]
I20260926 04:10:09.977593 14110 mvcc.cc:205] Tried to move back new op lower bound from 7333461237607428096 to 7333461237520289792. Current Snapshot: MvccSnapshot[applied={T|T < 7333461237607428096}]
I20260926 04:10:09.981375 13595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:09.981498 13595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:09.982354 13595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8637a4ae2ab246b696e246aff2a5af94: Sending DeleteTablet for 3 replicas of tablet abdedad8ee1a44f2ab3864d80161136c
I20260926 04:10:09.983086 13861 tablet_service.cc:1514] Processing DeleteTablet for tablet abdedad8ee1a44f2ab3864d80161136c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:09 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:56646
I20260926 04:10:09.983385 13727 tablet_service.cc:1514] Processing DeleteTablet for tablet abdedad8ee1a44f2ab3864d80161136c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:09 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:36664
I20260926 04:10:09.983441 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13633
I20260926 04:10:09.984436 13995 tablet_service.cc:1514] Processing DeleteTablet for tablet abdedad8ee1a44f2ab3864d80161136c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:09 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:47704
I20260926 04:10:09.988996 14118 tablet_replica.cc:330] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: stopping tablet replica
I20260926 04:10:09.989214 14119 tablet_replica.cc:330] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: stopping tablet replica
I20260926 04:10:09.989240 14118 raft_consensus.cc:2229] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:09.989354 14119 raft_consensus.cc:2229] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:10:09.989477 14119 raft_consensus.cc:2258] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:09.989708 14118 raft_consensus.cc:2258] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:09.991103 14118 ts_tablet_manager.cc:1905] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:09.993557 14118 ts_tablet_manager.cc:1918] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 04:10:09.996387 13579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:43405 (error 108)
I20260926 04:10:09.997398 14119 ts_tablet_manager.cc:1905] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:09.999297 14119 ts_tablet_manager.cc:1918] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:09.999401 14119 log.cc:1196] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/abdedad8ee1a44f2ab3864d80161136c
I20260926 04:10:09.999598 14118 log.cc:1196] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/abdedad8ee1a44f2ab3864d80161136c
I20260926 04:10:09.999691 14119 ts_tablet_manager.cc:1939] T abdedad8ee1a44f2ab3864d80161136c P f84ff8e80f3e4f44ac1f07e66c54a7b3: Deleting consensus metadata
I20260926 04:10:10.000119 14118 ts_tablet_manager.cc:1939] T abdedad8ee1a44f2ab3864d80161136c P 4573a72db6bb49698f903516cfa0730c: Deleting consensus metadata
I20260926 04:10:10.000288 13579 catalog_manager.cc:4841] TS f84ff8e80f3e4f44ac1f07e66c54a7b3 (127.0.213.195:35423): tablet abdedad8ee1a44f2ab3864d80161136c (table test-table [id=a428ac811a7441dabe76aef35dcb8569]) successfully deleted
I20260926 04:10:10.000705 13582 catalog_manager.cc:4841] TS 4573a72db6bb49698f903516cfa0730c (127.0.213.194:33291): tablet abdedad8ee1a44f2ab3864d80161136c (table test-table [id=a428ac811a7441dabe76aef35dcb8569]) successfully deleted
W20260926 04:10:10.002216 14093 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:10:10.002058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43405 (local address 127.0.0.1:36678)
0926 04:10:10.002115 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:10.002168 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43405: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 04:10:10.002367 13579 catalog_manager.cc:4584] TS 75d218f0c87c4ae39a44eaf8bb90a931 (127.0.213.193:43405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abdedad8ee1a44f2ab3864d80161136c: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43405: connect: Connection reset by peer (error 104)
I20260926 04:10:10.002398 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13797
I20260926 04:10:10.011495 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13930
I20260926 04:10:10.017117 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 13562
I20260926 04:10:10.023212 855 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 04:10:10.368897 855 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 04:10:10.369067 855 mini_ranger.cc:66] Stopping Ranger...
I20260926 04:10:10.714692 855 mini_ranger.cc:68] Stopped Ranger
2026-09-26 04:10:10.714 UTC [12629] LOG: received smart shutdown request
2026-09-26 04:10:10.719 UTC [12629] LOG: background worker "logical replication launcher" (PID 12638) exited with exit code 1
2026-09-26 04:10:10.719 UTC [12633] LOG: shutting down
2026-09-26 04:10:10.964 UTC [12629] LOG: database system is shut down
2026-09-26T04:10:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52061 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:10:11 dist-test-slave-xf24 krb5kdc[14129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:10:11 dist-test-slave-xf24 krb5kdc[14129](info): set up 2 sockets
Sep 26 04:10:11 dist-test-slave-xf24 krb5kdc[14129](info): commencing operation
krb5kdc: starting...
W20260926 04:10:12.172451 855 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:10:12.317474 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40009
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40009,127.0.213.253:43427,127.0.213.252:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:12.408393 14142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:12.408635 14142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:12.408660 14142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:12.410024 14142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:12.410102 14142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:12.410120 14142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:12.410133 14142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:12.410145 14142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:12.411890 14142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40009,127.0.213.253:43427,127.0.213.252:41131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:12.412252 14142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:12.412547 14142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:12.418066 14150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:12.418076 14148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:12.418121 14147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:12.418337 14142 server_base.cc:1034] running on GCE node
I20260926 04:10:12.418591 14142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:12.418924 14142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:12.420236 14142 hybrid_clock.cc:648] HybridClock initialized: now 1790395812420208 us; error 40 us; skew 500 ppm
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:12.426959 14142 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:12.427603 14142 webserver.cc:460] Webserver started at http://127.0.213.254:35711/ using document root <none> and password file <none>
I20260926 04:10:12.427911 14142 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:12.427995 14142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:12.428208 14142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:12.429486 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "49022721bc5a4bdba100fb2ed89c3968"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "5fa3c1043d6359b32fefcdef979f6313"
server_key_iv: "d00d3b7b899315c46a679d75b2c9a684"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.429927 14142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "49022721bc5a4bdba100fb2ed89c3968"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "5fa3c1043d6359b32fefcdef979f6313"
server_key_iv: "d00d3b7b899315c46a679d75b2c9a684"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.431536 14142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:10:12.432530 14157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:12.432916 14142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:12.433050 14142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "49022721bc5a4bdba100fb2ed89c3968"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "5fa3c1043d6359b32fefcdef979f6313"
server_key_iv: "d00d3b7b899315c46a679d75b2c9a684"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.433158 14142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:12.449414 14142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:12.452301 14142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:12.452436 14142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:12.457010 14142 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:40009
I20260926 04:10:12.457049 14209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:40009 every 8 connection(s)
I20260926 04:10:12.457440 14142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:12.458590 14210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:12.460425 14210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:10:12.463585 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14142
I20260926 04:10:12.463702 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:10:12.463966 855 external_mini_cluster.cc:1428] Setting key 7589eb2e1749739905c5e7c5bdb54939
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.253@KRBTEST.COM; defaulting to no policy
I20260926 04:10:12.472383 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.461289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.0.213.254:40009)
0926 04:10:12.461494 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.461504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.461680 (+ 176us) server_negotiation.cc:407] Connection header received
0926 04:10:12.462303 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.462306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.462324 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.462428 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.463029 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.463955 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.464922 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.465382 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.465543 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.468688 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.468698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.468698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.468715 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.470585 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.471041 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.471044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.471045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.471136 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.471365 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.471366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.471366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.471664 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.471755 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.471949 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.472119 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260926 04:10:12.472390 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.461208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40009 (local address 127.0.0.1:44016)
0926 04:10:12.461482 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.461572 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.462123 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.462433 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.462436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.462751 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.462911 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.462916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.464084 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.464085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.464815 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.464817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.465512 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.465513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.465629 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.466310 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.466319 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.468554 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.470721 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.470729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.470731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.470939 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.471208 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.471210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.471210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.471298 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.471744 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.471749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.471816 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.471925 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.472183 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":111,"thread_start_us":99,"threads_started":1}
I20260926 04:10:12.473553 14210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:10:12.474231 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.473913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32900)
0926 04:10:12.474019 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.474203 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 04:10:12.474426 14158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.253@KRBTEST.COM" created.
I20260926 04:10:12.474977 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.474861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32914)
0926 04:10:12.474911 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.474944 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 04:10:12.475082 14210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:43427: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab.
Entry for principal kudu/127.0.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab.
I20260926 04:10:12.491575 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:43427
--webserver_interface=127.0.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab
--principal=kudu/127.0.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40009,127.0.213.253:43427,127.0.213.252:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:12.524341 14210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } attempt: 1
I20260926 04:10:12.524947 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.524781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32930)
0926 04:10:12.524867 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.524914 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 04:10:12.525391 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.525280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32932)
0926 04:10:12.525338 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.525371 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 04:10:12.525493 14210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:43427: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
W20260926 04:10:12.574414 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:12.574705 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:12.574748 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:12.575965 14217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:12.576058 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:12.576084 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:12.576141 14217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:12.576164 14217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:12.577876 14217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40009,127.0.213.253:43427,127.0.213.252:41131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab
--principal=kudu/127.0.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:12.578183 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:12.578424 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:12.583933 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:12.583913 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:12.583913 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:12.584405 14217 server_base.cc:1034] running on GCE node
I20260926 04:10:12.584692 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:12.585047 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:12.586268 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790395812586240 us; error 46 us; skew 500 ppm
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:12.589208 14217 init.cc:377] Logged in from keytab as kudu/127.0.213.253@KRBTEST.COM (short username kudu)
I20260926 04:10:12.589761 14217 webserver.cc:460] Webserver started at http://127.0.213.253:33575/ using document root <none> and password file <none>
I20260926 04:10:12.590003 14217 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:12.590047 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:12.590183 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:12.591351 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data/instance:
uuid: "7a47f505b383448397bec402aa524fbc"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "94dc8809985ddc6823c3875dc9d423b6"
server_key_iv: "d5e5b0a5b0b63d2c77c926c21a9c7bdc"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.591733 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal/instance:
uuid: "7a47f505b383448397bec402aa524fbc"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "94dc8809985ddc6823c3875dc9d423b6"
server_key_iv: "d5e5b0a5b0b63d2c77c926c21a9c7bdc"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.593277 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:10:12.593670 14210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } attempt: 2
I20260926 04:10:12.594234 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.594016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32936)
0926 04:10:12.594150 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.594195 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 04:10:12.594254 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:12.594514 14217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:12.594581 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.594485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32942)
0926 04:10:12.594531 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.594556 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 04:10:12.594611 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal
uuid: "7a47f505b383448397bec402aa524fbc"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "94dc8809985ddc6823c3875dc9d423b6"
server_key_iv: "d5e5b0a5b0b63d2c77c926c21a9c7bdc"
server_key_version: "encryptionkey@0"
W20260926 04:10:12.594681 14210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:43427: Network error: Client connection negotiation failed: client connection to 127.0.213.253:43427: connect: Connection refused (error 111)
I20260926 04:10:12.594712 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:12.617673 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:12.620419 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:12.620582 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:12.624862 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.253:43427
I20260926 04:10:12.624938 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.253:43427 every 8 connection(s)
I20260926 04:10:12.625326 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
I20260926 04:10:12.626297 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:12.626811 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14217
I20260926 04:10:12.626947 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-1/wal/instance
I20260926 04:10:12.627161 855 external_mini_cluster.cc:1428] Setting key bef6a223b277f64209e9ad77e3fe099c
I20260926 04:10:12.628158 14285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.213.252@KRBTEST.COM; defaulting to no policy
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.252@KRBTEST.COM" created.
I20260926 04:10:12.638947 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.628962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.0.213.254:40009)
0926 04:10:12.629043 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.629045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.629311 (+ 266us) server_negotiation.cc:407] Connection header received
0926 04:10:12.629887 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.629888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.629902 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.629977 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.630697 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.631271 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.632659 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.632876 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.632952 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.635561 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.635563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.635563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.635571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.637255 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.637809 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.637811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.637812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.637889 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.638210 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.638210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.638211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.638333 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.638385 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.638794 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.638873 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:10:12.639235 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.628935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40009 (local address 127.0.0.1:44032)
0926 04:10:12.629146 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.629202 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.629786 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.630079 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.630082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.630368 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.630571 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.630576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.631383 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.631384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.632530 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.632533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.632972 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.632972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.633106 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.633847 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.633856 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.635438 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.637373 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.637381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.637384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.637732 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.637973 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.637974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.637974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.638137 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.638401 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.638407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.638597 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.638741 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.638932 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260926 04:10:12.640043 14285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:10:12.648365 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.640468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.0.213.253:43427)
0926 04:10:12.640630 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.640635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.640663 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:10:12.640683 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.640685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.640698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.640783 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.641062 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.641876 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.642594 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.642919 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.643011 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.644982 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.644993 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.644994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.645001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.646756 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.647255 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.647259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.647259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.647319 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.647598 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.647600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.647600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.647799 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.647917 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.648037 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.648233 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260926 04:10:12.648375 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.640404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32948)
0926 04:10:12.640466 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.640476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.640529 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.640783 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.640783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.640940 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.640987 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.640988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.641967 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.641969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.642496 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.642497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.643010 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.643011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.643081 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.643565 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.643571 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.644872 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.646879 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.646881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.646881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.647056 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.647409 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.647410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.647411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.647492 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.647889 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.647890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.647930 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.648000 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.648175 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 04:10:12.649386 14285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:10:12.649830 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.649658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42672)
0926 04:10:12.649702 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.649794 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 04:10:12.649974 14235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
I20260926 04:10:12.650445 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.650331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42678)
0926 04:10:12.650380 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.650414 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:10:12.650565 14285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:41131: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab.
Entry for principal kudu/127.0.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab.
I20260926 04:10:12.656766 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:41131
--webserver_interface=127.0.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab
--principal=kudu/127.0.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40009,127.0.213.253:43427,127.0.213.252:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:12.685848 14210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } attempt: 3
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:10:12.693679 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.686329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32962)
0926 04:10:12.686396 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.686406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.686472 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.686664 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.686665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.686835 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.686903 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.686905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.687621 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.687623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.688206 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.688208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.688588 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.688589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.688670 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.689373 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.689378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.690800 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.692504 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.692505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.692506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.692720 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.692936 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.692937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.692938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.693001 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.693318 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.693319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.693372 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.693462 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.693575 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 04:10:12.693679 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.686391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.213.253:43427)
0926 04:10:12.686456 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.686458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.686470 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:10:12.686552 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.686553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.686561 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.686609 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.686989 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.687524 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.688289 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.688503 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.688585 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.690912 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.690915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.690915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.690932 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.692411 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.692803 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.692804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.692805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.692866 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.693069 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.693070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.693070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.693242 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.693328 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.693488 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.693577 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:10:12.694344 14210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:10:12.694809 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.694682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42686)
0926 04:10:12.694742 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.694781 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:10:12.695303 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.695171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42694)
0926 04:10:12.695247 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.695280 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 04:10:12.695446 14210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:41131: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
I20260926 04:10:12.699790 14285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } attempt: 1
I20260926 04:10:12.700347 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.700208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42696)
0926 04:10:12.700271 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.700314 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 04:10:12.700820 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.700652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42706)
0926 04:10:12.700763 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.700794 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 04:10:12.700978 14285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:41131: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
I20260926 04:10:12.726658 14210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } attempt: 1
I20260926 04:10:12.727221 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.727039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42710)
0926 04:10:12.727149 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.727189 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 04:10:12.727646 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.727478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42724)
0926 04:10:12.727530 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.727602 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 04:10:12.727787 14210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:41131: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
W20260926 04:10:12.743681 14292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:12.744010 14292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:12.744084 14292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:12.745672 14292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:12.745759 14292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:12.745786 14292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:12.745813 14292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:12.745846 14292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:12.747687 14292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40009,127.0.213.253:43427,127.0.213.252:41131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab
--principal=kudu/127.0.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:12.748020 14292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:12.748323 14292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:12.753911 14297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:12.753903 14300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:12.753903 14298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:12.753935 14292 server_base.cc:1034] running on GCE node
I20260926 04:10:12.754531 14292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:12.754889 14292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:12.756131 14292 hybrid_clock.cc:648] HybridClock initialized: now 1790395812756119 us; error 37 us; skew 500 ppm
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:12.758900 14292 init.cc:377] Logged in from keytab as kudu/127.0.213.252@KRBTEST.COM (short username kudu)
I20260926 04:10:12.759440 14292 webserver.cc:460] Webserver started at http://127.0.213.252:44895/ using document root <none> and password file <none>
I20260926 04:10:12.759690 14292 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:12.759786 14292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:12.759948 14292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:12.761207 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data/instance:
uuid: "ca4529dca71b49328f4ee89de0f29a1b"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "587296672c735040aec5f82d233242af"
server_key_iv: "5901d6ed64a74ab3951adf78e9bcffd8"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.761581 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal/instance:
uuid: "ca4529dca71b49328f4ee89de0f29a1b"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "587296672c735040aec5f82d233242af"
server_key_iv: "5901d6ed64a74ab3951adf78e9bcffd8"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.763021 14292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:10:12.763954 14307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:12.764240 14292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:12.764353 14292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal
uuid: "ca4529dca71b49328f4ee89de0f29a1b"
format_stamp: "Formatted at 2026-09-26 04:10:12 on dist-test-slave-xf24"
server_key: "587296672c735040aec5f82d233242af"
server_key_iv: "5901d6ed64a74ab3951adf78e9bcffd8"
server_key_version: "encryptionkey@0"
I20260926 04:10:12.764449 14292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:12.769155 14285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } attempt: 2
I20260926 04:10:12.769619 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.769471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42738)
0926 04:10:12.769535 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.769583 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:10:12.769989 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.769893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42748)
0926 04:10:12.769940 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.769968 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 04:10:12.770108 14285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:41131: Network error: Client connection negotiation failed: client connection to 127.0.213.252:41131: connect: Connection refused (error 111)
I20260926 04:10:12.781744 14292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:12.784595 14292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:12.784781 14292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:12.789186 14292 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.252:41131
I20260926 04:10:12.789211 14359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.252:41131 every 8 connection(s)
I20260926 04:10:12.789660 14292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
I20260926 04:10:12.790642 14360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:12.792161 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14292
I20260926 04:10:12.792285 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395654118158-855-0/minicluster-data/master-2/wal/instance
I20260926 04:10:12.792500 855 external_mini_cluster.cc:1428] Setting key 7258bc4d06597a6a84efd20709186885
I20260926 04:10:12.792479 14360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
I20260926 04:10:12.802943 14210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } attempt: 2
I20260926 04:10:12.804832 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.793444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.0.213.254:40009)
0926 04:10:12.793504 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.793506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.793665 (+ 159us) server_negotiation.cc:407] Connection header received
0926 04:10:12.794122 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.794123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.794135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.794193 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.795023 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.795811 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.797244 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.797567 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.797647 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.801362 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.801364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.801364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.801375 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.803037 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.803761 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.803763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.803763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.803824 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.804125 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.804126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.804126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.804276 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.804342 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.804650 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.804740 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 04:10:12.805060 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.793357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40009 (local address 127.0.0.1:44044)
0926 04:10:12.793530 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.793569 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.794001 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.794296 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.794299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.794684 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.794928 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.794934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.795897 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.795898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.797101 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.797104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.797722 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.797724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.797869 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.798623 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.798634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.801202 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.803166 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.803174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.803177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.803647 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.803899 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.803900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.803900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.804043 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.804343 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.804350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.804512 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.804614 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.804767 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260926 04:10:12.805428 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.793188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.0.213.252:41131)
0926 04:10:12.793381 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.793391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.793430 (+ 39us) server_negotiation.cc:407] Connection header received
0926 04:10:12.793963 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.793971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.793992 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.794123 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.794941 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.796171 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.797227 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.797803 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.797868 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.801565 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.801573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.801576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.801584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.803775 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.804281 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.804285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.804286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.804342 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.804538 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.804539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.804539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.804706 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.804838 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.805254 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.805330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260926 04:10:12.806205 14360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:10:12.811638 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.803278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42758)
0926 04:10:12.803338 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.803350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.803389 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.803705 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.803706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.803870 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.803929 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.803930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.804928 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.804929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.805653 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.805656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.806328 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.806329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.806387 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.806899 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.806904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.808456 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.810467 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.810468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.810468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.810724 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.810947 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.810948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.810948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.811017 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.811350 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.811351 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.811396 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.811448 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.811536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:12.812264 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.803344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.0.213.252:41131)
0926 04:10:12.803560 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.803562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.803569 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:10:12.803590 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.803590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.803604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.803681 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.804012 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.804833 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.805969 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.806271 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.806331 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.808549 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.808550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.808551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.808569 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.810353 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.810808 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.810810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.810810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.810875 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.811103 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.811104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.811104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.811256 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.811999 (+ 743us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.812044 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.812145 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260926 04:10:12.813988 14210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: Bootstrap starting.
I20260926 04:10:12.814482 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.806544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.0.213.253:43427)
0926 04:10:12.806588 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.806590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.806600 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:12.806649 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.806650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.806658 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.806706 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.807067 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.807714 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.808503 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.808712 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.808804 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.811199 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.811200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.811200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.811208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.813270 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.813636 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.813638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.813638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.813700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.813913 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.813914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.813914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.814026 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.814116 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.814343 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.814420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:10:12.814746 14210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:12.814777 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.806469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:43427 (local address 127.0.0.1:32968)
0926 04:10:12.806540 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.806552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.806590 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.806748 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.806749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.806911 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.806964 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.806966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.807792 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.807792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.808423 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.808425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.808802 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.808803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.808869 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.809731 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.809736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.811104 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.813372 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.813373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.813374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.813544 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.813770 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.813771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.813771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.813828 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.814144 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.814144 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.814206 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.814579 (+ 373us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.814688 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 04:10:12.815090 14210 log.cc:824] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:12.815528 14360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:10:12.816545 14210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: No bootstrap required, opened a new log
I20260926 04:10:12.818753 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.806542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.0.213.254:40009)
0926 04:10:12.806622 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.806624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.806699 (+ 75us) server_negotiation.cc:407] Connection header received
0926 04:10:12.806786 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.806787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.806796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.806844 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.807302 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.807952 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.809748 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.810045 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.810103 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.815577 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.815579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.815579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.815587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.817358 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.817774 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.817776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.817776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.817840 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.818054 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.818055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.818056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.818194 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.818341 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.818561 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.818661 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 04:10:12.818878 14210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:12.819056 14210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:12.819123 14210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49022721bc5a4bdba100fb2ed89c3968, State: Initialized, Role: FOLLOWER
I20260926 04:10:12.819285 14210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:12.819604 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:12.819756 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:10:12.819682 14210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
W20260926 04:10:12.823062 14378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:10:12.823201 14378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:10:12.823849 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.815860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42766)
0926 04:10:12.815942 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.815951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.815999 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.816186 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.816187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.816340 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.816378 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.816380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.817312 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.817313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.818115 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.818122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.818511 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.818512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.818568 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.819216 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.819222 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.820814 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.822772 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.822774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.822774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.823006 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.823219 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.823220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.823220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.823282 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.823558 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.823559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.823618 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.823691 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.823781 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 04:10:12.823943 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.815890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.0.213.252:41131)
0926 04:10:12.815934 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.815937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.815988 (+ 51us) server_negotiation.cc:407] Connection header received
0926 04:10:12.816092 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.816093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.816104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.816149 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.816725 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.817220 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.818201 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.818430 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.818504 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.820929 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.820930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.820931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.820939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.822689 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.823091 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.823092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.823092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.823152 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.823340 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.823342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.823342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.823487 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.823565 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.823745 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.823855 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":21}
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:10:12.825825 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b: Bootstrap starting.
I20260926 04:10:12.822918 14142 master_runner.cc:186] Error getting master registration for 127.0.213.253:43427: OK, instance_id { permanent_uuid: "7a47f505b383448397bec402aa524fbc" instance_seqno: 1790395812623600 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:10:12.826231 14142 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:10:12.826545 14360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:12.826903 14360 log.cc:824] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:12.828976 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b: No bootstrap required, opened a new log
I20260926 04:10:12.830806 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.819810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.0.213.253:43427)
0926 04:10:12.819856 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.819859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.819867 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:10:12.819896 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.819897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.819907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.819966 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.820665 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.821724 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.822639 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.822951 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.823024 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.826311 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.826313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.826314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.826323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.828833 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.829376 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.829378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.829379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.829454 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.829753 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.829755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.829755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.829908 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.830567 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.830621 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.830709 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:10:12.831319 14360 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:12.831523 14360 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:12.831568 14360 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4529dca71b49328f4ee89de0f29a1b, State: Initialized, Role: FOLLOWER
I20260926 04:10:12.831771 14360 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:12.832119 14381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:12.832212 14360 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:12.832280 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:10:12.832281 14381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:10:12.837234 14396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:10:12.837399 14396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:10:12.836440 14292 master_runner.cc:186] Error getting master registration for 127.0.213.253:43427: OK, instance_id { permanent_uuid: "7a47f505b383448397bec402aa524fbc" instance_seqno: 1790395812623600 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:10:12.840243 14292 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:10:12.843576 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.834102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.0.213.253:43427)
0926 04:10:12.834138 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.834140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.834262 (+ 122us) server_negotiation.cc:407] Connection header received
0926 04:10:12.834365 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.834366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.834377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.834411 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.835325 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.836106 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.837572 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.837874 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.837950 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.840473 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.840475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.840475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.840484 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.842183 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.842576 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.842577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.842577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.842629 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.842836 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.842837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.842838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.843016 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.843092 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.843302 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.843473 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 04:10:12.848260 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.833896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.213.254:40009)
0926 04:10:12.833940 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.833942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.834381 (+ 439us) server_negotiation.cc:407] Connection header received
0926 04:10:12.834496 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.834497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.834509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.834539 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.835353 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.838705 (+ 3352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.842043 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.842609 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.842670 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.845090 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.845091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.845092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.845100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.846852 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.847275 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.847277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.847278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.847337 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.847567 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.847568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.847569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.847717 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.847810 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.848040 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.848157 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:10:12.850097 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.836393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.213.252:41131)
0926 04:10:12.836431 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.836433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.837642 (+ 1209us) server_negotiation.cc:407] Connection header received
0926 04:10:12.837735 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.837736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.837746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.837775 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.838575 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.839322 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.840307 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.840606 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.840650 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.842041 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.842043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.842043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.842051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.843730 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.849218 (+ 5488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.849219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.849220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.849301 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.849516 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.849518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.849518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.849684 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.849804 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.849917 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.850004 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:10:12.861234 14285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } attempt: 3
Sep 26 04:10:12 dist-test-slave-xf24 krb5kdc[14129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
I20260926 04:10:12.870221 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.861686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:41131 (local address 127.0.0.1:42788)
0926 04:10:12.861774 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:12.861786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:12.861827 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:12.862066 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:12.862067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:12.862273 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:12.862326 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.862328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.863280 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.863281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.864137 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:12.864140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.864622 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.864623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:12.864681 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.865227 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:12.865233 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:12.866872 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:12.869002 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.869004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.869005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.869252 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.869508 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:12.869509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:12.869509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:12.869574 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:12.869875 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:12.869876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:12.869928 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 04:10:12.869991 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:10:12.870100 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 04:10:12.870905 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:12.861741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.0.213.252:41131)
0926 04:10:12.861789 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:12.861791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:12.861935 (+ 144us) server_negotiation.cc:407] Connection header received
0926 04:10:12.861962 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:12.861962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:12.861973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:12.862020 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:12.862422 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.863182 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.864236 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:12.864543 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:12.864615 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:12.866965 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:12.866967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:12.866967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:12.866976 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:12.868904 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.869350 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.869351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.869352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.869420 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:12.869643 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:12.869643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:12.869644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:12.869787 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:12.869849 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:12.870725 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 04:10:12.870813 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:10:12.872609 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: Bootstrap starting.
I20260926 04:10:12.873715 14285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:12.874148 14285 log.cc:824] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:12.875092 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: No bootstrap required, opened a new log
I20260926 04:10:12.877215 14285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:12.877440 14285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:12.877524 14285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a47f505b383448397bec402aa524fbc, State: Initialized, Role: FOLLOWER
I20260926 04:10:12.877727 14285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:12.878150 14285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:12.878109 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:12.878353 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:10:12.880118 14413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:10:12.880224 14413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:10:12.882381 14217 master_runner.cc:263] No leader master found from master 7a47f505b383448397bec402aa524fbc
I20260926 04:10:12.882438 14217 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 04:10:12.953531 14163 tablet.cc:2369] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:10:13.022442 14381 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:13.022590 14381 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:13.022854 14381 leader_election.cc:288] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49022721bc5a4bdba100fb2ed89c3968 (127.0.213.254:40009), 7a47f505b383448397bec402aa524fbc (127.0.213.253:43427)
I20260926 04:10:13.023154 14184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4529dca71b49328f4ee89de0f29a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49022721bc5a4bdba100fb2ed89c3968" is_pre_election: true
I20260926 04:10:13.023233 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4529dca71b49328f4ee89de0f29a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a47f505b383448397bec402aa524fbc" is_pre_election: true
I20260926 04:10:13.023337 14184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4529dca71b49328f4ee89de0f29a1b in term 0.
I20260926 04:10:13.023367 14259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4529dca71b49328f4ee89de0f29a1b in term 0.
I20260926 04:10:13.023542 14308 leader_election.cc:302] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a47f505b383448397bec402aa524fbc, ca4529dca71b49328f4ee89de0f29a1b; no voters:
I20260926 04:10:13.023717 14381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:10:13.023761 14381 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:10:13.023782 14381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:13.024746 14381 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:13.025084 14184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4529dca71b49328f4ee89de0f29a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49022721bc5a4bdba100fb2ed89c3968"
I20260926 04:10:13.025373 14184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:13.026304 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4529dca71b49328f4ee89de0f29a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a47f505b383448397bec402aa524fbc"
I20260926 04:10:13.026397 14259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:13.026445 14184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4529dca71b49328f4ee89de0f29a1b in term 1.
I20260926 04:10:13.026803 14311 leader_election.cc:302] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49022721bc5a4bdba100fb2ed89c3968, ca4529dca71b49328f4ee89de0f29a1b; no voters:
I20260926 04:10:13.027011 14415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:13.027132 14415 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [term 1 LEADER]: Becoming Leader. State: Replica: ca4529dca71b49328f4ee89de0f29a1b, State: Running, Role: LEADER
I20260926 04:10:13.027237 14415 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } }
I20260926 04:10:13.027272 14259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4529dca71b49328f4ee89de0f29a1b in term 1.
I20260926 04:10:13.027724 14381 leader_election.cc:288] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [CANDIDATE]: Term 1 election: Requested vote from peers 49022721bc5a4bdba100fb2ed89c3968 (127.0.213.254:40009), 7a47f505b383448397bec402aa524fbc (127.0.213.253:43427)
I20260926 04:10:13.027797 14415 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4529dca71b49328f4ee89de0f29a1b. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.027904 14415 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:13.028206 14418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:13.028463 14418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:13.030318 14259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [term 1 FOLLOWER]: Refusing update from remote peer ca4529dca71b49328f4ee89de0f29a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:13.031097 14184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [term 1 FOLLOWER]: Refusing update from remote peer ca4529dca71b49328f4ee89de0f29a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:13.031443 14416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:13.031625 14415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:13.032927 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4529dca71b49328f4ee89de0f29a1b. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.033041 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:10:13.033670 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4529dca71b49328f4ee89de0f29a1b. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.033772 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:10:13.034458 14419 mvcc.cc:205] Tried to move back new op lower bound from 7333461250169352192 to 7333461250160308224. Current Snapshot: MvccSnapshot[applied={T|T < 7333461250169352192}]
I20260926 04:10:13.034565 14381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.034657 14381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:13.034603 14415 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.034806 14415 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:13.034960 14402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.035070 14402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:10:13.035240 14422 mvcc.cc:205] Tried to move back new op lower bound from 7333461250169352192 to 7333461250160308224. Current Snapshot: MvccSnapshot[applied={T|T < 7333461250169352192}]
I20260926 04:10:13.035421 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca4529dca71b49328f4ee89de0f29a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49022721bc5a4bdba100fb2ed89c3968" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40009 } } peers { permanent_uuid: "7a47f505b383448397bec402aa524fbc" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 43427 } } peers { permanent_uuid: "ca4529dca71b49328f4ee89de0f29a1b" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 41131 } } }
I20260926 04:10:13.035516 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:10:13.035655 14418 catalog_manager.cc:1347] Generated new cluster ID: 743ff202bdde45e38eff886e5878073b
I20260926 04:10:13.035710 14418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:13.049858 14418 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:10:13.050973 14418 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:13.058851 14418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca4529dca71b49328f4ee89de0f29a1b: Generated new TSK 0
I20260926 04:10:13.059185 14418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:10:13.823736 14378 catalog_manager.cc:1259] Loaded cluster ID: 743ff202bdde45e38eff886e5878073b
I20260926 04:10:13.823824 14378 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: loading cluster ID for follower catalog manager: success
I20260926 04:10:13.824692 14378 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: acquiring CA information for follower catalog manager: success
I20260926 04:10:13.824990 14378 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49022721bc5a4bdba100fb2ed89c3968: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:10:13.880738 14413 catalog_manager.cc:1259] Loaded cluster ID: 743ff202bdde45e38eff886e5878073b
I20260926 04:10:13.880827 14413 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: loading cluster ID for follower catalog manager: success
I20260926 04:10:13.881855 14413 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: acquiring CA information for follower catalog manager: success
I20260926 04:10:13.882176 14413 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a47f505b383448397bec402aa524fbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:10:14.120009 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14142
I20260926 04:10:14.125794 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14217
I20260926 04:10:14.130985 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14292
2026-09-26T04:10:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3054 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:10:14 dist-test-slave-xf24 krb5kdc[14433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:10:14 dist-test-slave-xf24 krb5kdc[14433](info): set up 2 sockets
Sep 26 04:10:14 dist-test-slave-xf24 krb5kdc[14433](info): commencing operation
krb5kdc: starting...
W20260926 04:10:15.188171 855 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:10:15 dist-test-slave-xf24 krb5kdc[14433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:10:15.316340 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41983
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:41983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:15.402290 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:15.402529 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:15.402554 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:15.403760 14446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:15.403803 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:15.403817 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:15.403831 14446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:15.403842 14446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:15.405423 14446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:41983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:15.405640 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:15.405823 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:15.411088 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:15.411088 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:15.411177 14446 server_base.cc:1034] running on GCE node
W20260926 04:10:15.411309 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:15.411568 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:15.411880 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:15.413051 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790395815413045 us; error 43 us; skew 500 ppm
Sep 26 04:10:15 dist-test-slave-xf24 krb5kdc[14433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:15.415663 14446 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:15.416164 14446 webserver.cc:460] Webserver started at http://127.0.213.254:41981/ using document root <none> and password file <none>
I20260926 04:10:15.416420 14446 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:15.416486 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:15.416635 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:15.417812 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "50e0d54c800f4d238f6da780774ac687"
format_stamp: "Formatted at 2026-09-26 04:10:15 on dist-test-slave-xf24"
server_key: "201bb3ed4c74f64b2c52ef3fc4cc70f4"
server_key_iv: "9e767c25f6ed56cd594f1feb0e863fcc"
server_key_version: "encryptionkey@0"
I20260926 04:10:15.418192 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "50e0d54c800f4d238f6da780774ac687"
format_stamp: "Formatted at 2026-09-26 04:10:15 on dist-test-slave-xf24"
server_key: "201bb3ed4c74f64b2c52ef3fc4cc70f4"
server_key_iv: "9e767c25f6ed56cd594f1feb0e863fcc"
server_key_version: "encryptionkey@0"
I20260926 04:10:15.419761 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:10:15.421196 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:15.421403 14446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:15.421521 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "50e0d54c800f4d238f6da780774ac687"
format_stamp: "Formatted at 2026-09-26 04:10:15 on dist-test-slave-xf24"
server_key: "201bb3ed4c74f64b2c52ef3fc4cc70f4"
server_key_iv: "9e767c25f6ed56cd594f1feb0e863fcc"
server_key_version: "encryptionkey@0"
I20260926 04:10:15.421607 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:15.450232 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:15.452770 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:15.452948 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:15.457998 14446 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:41983
I20260926 04:10:15.458031 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:41983 every 8 connection(s)
I20260926 04:10:15.458441 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:15.459431 14514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:15.461959 14514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687: Bootstrap starting.
I20260926 04:10:15.462026 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14446
I20260926 04:10:15.462113 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:10:15.462328 855 external_mini_cluster.cc:1428] Setting key 0a3199c7665edc610678c515eee65ade
I20260926 04:10:15.462842 14514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:15.463191 14514 log.cc:824] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:15.463920 14514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687: No bootstrap required, opened a new log
I20260926 04:10:15.465863 14514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54c800f4d238f6da780774ac687" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41983 } }
I20260926 04:10:15.466048 14514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:15.466099 14514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e0d54c800f4d238f6da780774ac687, State: Initialized, Role: FOLLOWER
I20260926 04:10:15.466239 14514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54c800f4d238f6da780774ac687" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41983 } }
I20260926 04:10:15.466336 14514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:15.466383 14514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:15.466432 14514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:15.467095 14514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54c800f4d238f6da780774ac687" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41983 } }
I20260926 04:10:15.467218 14514 leader_election.cc:302] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50e0d54c800f4d238f6da780774ac687; no voters:
I20260926 04:10:15.467428 14514 leader_election.cc:288] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:10:15.467541 14519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:15.467718 14514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:15.467724 14519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [term 1 LEADER]: Becoming Leader. State: Replica: 50e0d54c800f4d238f6da780774ac687, State: Running, Role: LEADER
I20260926 04:10:15.467978 14519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54c800f4d238f6da780774ac687" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41983 } }
I20260926 04:10:15.468542 14521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50e0d54c800f4d238f6da780774ac687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54c800f4d238f6da780774ac687" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41983 } } }
I20260926 04:10:15.468626 14521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:15.468786 14520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50e0d54c800f4d238f6da780774ac687. Latest consensus state: current_term: 1 leader_uuid: "50e0d54c800f4d238f6da780774ac687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e0d54c800f4d238f6da780774ac687" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41983 } } }
I20260926 04:10:15.468854 14520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687 [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:15.469435 14529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 04:10:15 dist-test-slave-xf24 krb5kdc[14433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:15.470088 14529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:15.471994 14529 catalog_manager.cc:1347] Generated new cluster ID: fa1df7c6f3aa4a32a56707aca85896e9
I20260926 04:10:15.472050 14529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:15.474020 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:15.462923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.213.254:41983)
0926 04:10:15.463179 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:15.463187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:15.463215 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:10:15.463718 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:15.463725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:15.463782 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:15.463951 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:15.464643 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:15.465545 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:15.466438 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:15.466912 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:15.467034 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:15.470393 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:15.470398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:15.470399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:15.470412 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:15.472452 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:15.472841 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:15.472844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:15.472845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:15.472942 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:15.473153 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:15.473154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:15.473154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:15.473402 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:15.473568 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:15.473679 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 04:10:15.473833 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
I20260926 04:10:15.474936 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:10:15.491328 14529 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:10:15.491992 14529 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:15.502396 14529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50e0d54c800f4d238f6da780774ac687: Generated new TSK 0
I20260926 04:10:15.502650 14529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:10:15.529209 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14446
2026-09-26T04:10:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1396 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:10:15 dist-test-slave-xf24 krb5kdc[14541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:10:15 dist-test-slave-xf24 krb5kdc[14541](info): set up 2 sockets
Sep 26 04:10:15 dist-test-slave-xf24 krb5kdc[14541](info): commencing operation
krb5kdc: starting...
W20260926 04:10:16.600394 855 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:10:16 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:10:16.721530 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45691
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:16.803704 14554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:16.803938 14554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:16.803961 14554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:16.805171 14554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:16.805219 14554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:16.805234 14554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:16.805246 14554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:16.805259 14554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:16.806701 14554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:16.806910 14554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:16.807085 14554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:16.811863 14559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:16.811851 14562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:16.811849 14560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:16.812218 14554 server_base.cc:1034] running on GCE node
I20260926 04:10:16.812461 14554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:16.812706 14554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:16.813863 14554 hybrid_clock.cc:648] HybridClock initialized: now 1790395816813852 us; error 32 us; skew 500 ppm
Sep 26 04:10:16 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:16.816630 14554 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:16.817184 14554 webserver.cc:460] Webserver started at http://127.0.213.254:46099/ using document root <none> and password file <none>
I20260926 04:10:16.817428 14554 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:16.817499 14554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:16.817653 14554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:16.818809 14554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "36eb6a2c5be644b9b207bcb2227e11bc"
format_stamp: "Formatted at 2026-09-26 04:10:16 on dist-test-slave-xf24"
server_key: "700b79344b4559c3b1feb6e80a637e14"
server_key_iv: "2a7dcd0c63c501f3b0fd0297177ce01b"
server_key_version: "encryptionkey@0"
I20260926 04:10:16.819206 14554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "36eb6a2c5be644b9b207bcb2227e11bc"
format_stamp: "Formatted at 2026-09-26 04:10:16 on dist-test-slave-xf24"
server_key: "700b79344b4559c3b1feb6e80a637e14"
server_key_iv: "2a7dcd0c63c501f3b0fd0297177ce01b"
server_key_version: "encryptionkey@0"
I20260926 04:10:16.820848 14554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:10:16.821743 14569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:16.822024 14554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:16.822150 14554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "36eb6a2c5be644b9b207bcb2227e11bc"
format_stamp: "Formatted at 2026-09-26 04:10:16 on dist-test-slave-xf24"
server_key: "700b79344b4559c3b1feb6e80a637e14"
server_key_iv: "2a7dcd0c63c501f3b0fd0297177ce01b"
server_key_version: "encryptionkey@0"
I20260926 04:10:16.822229 14554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:16.840420 14554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:16.842528 14554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:16.842691 14554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:16.847028 14554 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45691
I20260926 04:10:16.847071 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45691 every 8 connection(s)
I20260926 04:10:16.847482 14554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:16.848564 14622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:16.850903 14622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap starting.
I20260926 04:10:16.851532 14622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:16.851826 14622 log.cc:824] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:16.852730 14622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: No bootstrap required, opened a new log
I20260926 04:10:16.854449 14622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:16.854630 14622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:16.854678 14622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36eb6a2c5be644b9b207bcb2227e11bc, State: Initialized, Role: FOLLOWER
I20260926 04:10:16.854851 14622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:16.854933 14622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:16.854976 14622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:16.855028 14622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:16.855906 14622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:16.856029 14622 leader_election.cc:302] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36eb6a2c5be644b9b207bcb2227e11bc; no voters:
I20260926 04:10:16.856231 14622 leader_election.cc:288] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:10:16.856343 14625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:16.856496 14622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:16.856521 14625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 LEADER]: Becoming Leader. State: Replica: 36eb6a2c5be644b9b207bcb2227e11bc, State: Running, Role: LEADER
I20260926 04:10:16.856720 14625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:16.857128 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14554
I20260926 04:10:16.857124 14626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } } }
I20260926 04:10:16.857136 14627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36eb6a2c5be644b9b207bcb2227e11bc. Latest consensus state: current_term: 1 leader_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } } }
I20260926 04:10:16.857234 14626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:16.857246 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:10:16.857255 14627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:16.857467 855 external_mini_cluster.cc:1428] Setting key 5a21531e616f73e99bd49cc22049543e
I20260926 04:10:16.857488 14634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:16.858057 14634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:16.859977 14634 catalog_manager.cc:1347] Generated new cluster ID: d00dc68c372b4648a172174e478f1c3a
I20260926 04:10:16.860105 14634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:10:16 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:16.870571 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:16.858218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.213.254:45691)
0926 04:10:16.858405 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:16.858416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:16.858569 (+ 153us) server_negotiation.cc:407] Connection header received
0926 04:10:16.858821 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:16.858830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:16.858899 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:16.859079 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:16.859956 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:16.861536 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:16.862700 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:16.863377 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:16.863507 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:16.866539 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:16.866544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:16.866544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:16.866559 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:16.868792 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:16.869232 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:16.869242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:16.869243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:16.869362 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:16.869576 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:16.869577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:16.869577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:16.869905 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:16.870065 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:16.870165 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 04:10:16.870320 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260926 04:10:16.873265 14634 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:10:16.873788 14634 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:16.878340 14634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Generated new TSK 0
I20260926 04:10:16.878525 14634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:10:16.911576 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45691
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:16.996518 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:16.996737 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:16.996762 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:16.997942 14648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:16.997987 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:16.998015 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:16.999493 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:16.999751 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:17.000010 14648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:17.005265 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.005261 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.005342 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.005745 14648 server_base.cc:1034] running on GCE node
I20260926 04:10:17.005982 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:17.006280 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:17.007467 14648 hybrid_clock.cc:648] HybridClock initialized: now 1790395817007445 us; error 38 us; skew 500 ppm
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:17.010149 14648 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:17.010687 14648 webserver.cc:460] Webserver started at http://127.0.213.193:34293/ using document root <none> and password file <none>
I20260926 04:10:17.010962 14648 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:17.011032 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:17.011178 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:17.012324 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "138fa16b109640ed88f6d901df885755"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "0f3178d6eb203cb87068525dadb3a89c"
server_key_iv: "ef9849ecd929ffae4b5c3cdbe63917a9"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.012698 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "138fa16b109640ed88f6d901df885755"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "0f3178d6eb203cb87068525dadb3a89c"
server_key_iv: "ef9849ecd929ffae4b5c3cdbe63917a9"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.014045 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:10:17.014802 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.014987 14648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.015089 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "138fa16b109640ed88f6d901df885755"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "0f3178d6eb203cb87068525dadb3a89c"
server_key_iv: "ef9849ecd929ffae4b5c3cdbe63917a9"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.015164 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:17.029201 14648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:17.031579 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:17.031772 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:17.032047 14648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:17.032501 14648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:17.032558 14648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.032604 14648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:17.032646 14648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.039407 14648 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:34569
I20260926 04:10:17.039465 14776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:34569 every 8 connection(s)
I20260926 04:10:17.039839 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:17.046710 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14648
I20260926 04:10:17.046824 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:10:17.047050 855 external_mini_cluster.cc:1428] Setting key 251b52fcc10a16925a427877879982b6
I20260926 04:10:17.050434 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.040767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:40063 (local address 127.0.213.254:45691)
0926 04:10:17.040842 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.040844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.041058 (+ 214us) server_negotiation.cc:407] Connection header received
0926 04:10:17.041548 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.041549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.041563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.041631 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:17.042648 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.043184 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.044254 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.044470 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.044553 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.047056 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:17.047057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:17.047057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:17.047065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:17.048695 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.049283 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.049285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.049286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.049365 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.049668 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.049669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.049670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.049829 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:17.049939 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.050264 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.050363 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 04:10:17.050607 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.040685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:40063)
0926 04:10:17.040882 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.040928 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.041451 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.041688 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.041692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.042331 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:17.042543 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.042549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.043303 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.043304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.044161 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.044164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.044575 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.044576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.044693 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.045243 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:17.045251 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:17.046954 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:17.048784 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.048792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.048794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.049172 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.049473 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.049474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.049474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.049594 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.049960 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:17.049966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:17.050107 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.050224 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.050375 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 04:10:17.051195 14777 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45691
I20260926 04:10:17.051307 14777 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:17.051510 14777 heartbeater.cc:511] Master 127.0.213.254:45691 requested a full tablet report, sending...
I20260926 04:10:17.052206 14586 ts_manager.cc:194] Registered new tserver with Master: 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569)
I20260926 04:10:17.052850 14586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:40063
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:10:17.077049 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45691
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:17.159984 14782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:17.160238 14782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:17.160264 14782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:17.161511 14782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:17.161561 14782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:17.161589 14782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:17.163057 14782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:17.163280 14782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:17.163492 14782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:17.168633 14790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.168659 14788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.168949 14782 server_base.cc:1034] running on GCE node
W20260926 04:10:17.168633 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.169273 14782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:17.169525 14782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:17.170694 14782 hybrid_clock.cc:648] HybridClock initialized: now 1790395817170683 us; error 29 us; skew 500 ppm
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:17.173501 14782 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:17.173995 14782 webserver.cc:460] Webserver started at http://127.0.213.194:40589/ using document root <none> and password file <none>
I20260926 04:10:17.174256 14782 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:17.174328 14782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:17.174477 14782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:17.175616 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "633a007ae0074c9ca56072892b2e6525"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "28fbfd25dc3346a87c5ec4c1abc9ae58"
server_key_iv: "ea471847c35209163fc99b87fe406c9f"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.175999 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "633a007ae0074c9ca56072892b2e6525"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "28fbfd25dc3346a87c5ec4c1abc9ae58"
server_key_iv: "ea471847c35209163fc99b87fe406c9f"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.177485 14782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.178265 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.178475 14782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.178583 14782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "633a007ae0074c9ca56072892b2e6525"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "28fbfd25dc3346a87c5ec4c1abc9ae58"
server_key_iv: "ea471847c35209163fc99b87fe406c9f"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.178660 14782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:17.196038 14782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:17.198163 14782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:17.198331 14782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:17.198566 14782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:17.198892 14782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:17.198945 14782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.198987 14782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:17.199015 14782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.204946 14782 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:45791
I20260926 04:10:17.205000 14910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:45791 every 8 connection(s)
I20260926 04:10:17.205348 14782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:17.212356 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14782
I20260926 04:10:17.212457 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:10:17.212699 855 external_mini_cluster.cc:1428] Setting key 02d1d70ff6196c825674eeeb81e38472
I20260926 04:10:17.215576 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.206064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:50609 (local address 127.0.213.254:45691)
0926 04:10:17.206125 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.206127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.206380 (+ 253us) server_negotiation.cc:407] Connection header received
0926 04:10:17.206884 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.206885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.206895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.206949 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:17.207899 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.208543 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.209591 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.209809 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.209883 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.212203 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:17.212205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:17.212205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:17.212215 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:17.214141 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.214573 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.214575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.214575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.214641 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.214868 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.214869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.214870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.215024 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:17.215098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.215446 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.215514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 04:10:17.215914 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.206058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:50609)
0926 04:10:17.206202 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.206297 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.206783 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.207001 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.207005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.207588 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:17.207782 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.207786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.208643 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.208644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.209501 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.209504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.209908 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.209909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.210016 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.210548 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:17.210557 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:17.212035 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:17.214221 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.214260 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.214262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.214499 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.214709 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.214709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.214709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.214809 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.215108 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:17.215113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:17.215287 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.215427 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.215615 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260926 04:10:17.216487 14911 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45691
I20260926 04:10:17.216615 14911 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:17.216841 14911 heartbeater.cc:511] Master 127.0.213.254:45691 requested a full tablet report, sending...
I20260926 04:10:17.217321 14586 ts_manager.cc:194] Registered new tserver with Master: 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791)
I20260926 04:10:17.217778 14586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:50609
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:10:17.242645 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45691
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:17.325464 14916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:17.325732 14916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:17.325775 14916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:17.327006 14916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:17.327075 14916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:17.327121 14916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:17.328708 14916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:17.328990 14916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:17.329227 14916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:17.334293 14924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.334270 14921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.334321 14922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.334635 14916 server_base.cc:1034] running on GCE node
I20260926 04:10:17.334844 14916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:17.335065 14916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:17.336199 14916 hybrid_clock.cc:648] HybridClock initialized: now 1790395817336186 us; error 29 us; skew 500 ppm
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:17.338966 14916 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:17.339489 14916 webserver.cc:460] Webserver started at http://127.0.213.195:44263/ using document root <none> and password file <none>
I20260926 04:10:17.339727 14916 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:17.339779 14916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:17.339955 14916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:17.341151 14916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "d491ccf7c4ee416fb946982438e13f17"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "65d775332f1aec0d79173568ceb89d34"
server_key_iv: "ad29afd1d5ed0cd916357661d5037d2a"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.341544 14916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "d491ccf7c4ee416fb946982438e13f17"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "65d775332f1aec0d79173568ceb89d34"
server_key_iv: "ad29afd1d5ed0cd916357661d5037d2a"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.343003 14916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:10:17.343887 14931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.344141 14916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:17.344236 14916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "d491ccf7c4ee416fb946982438e13f17"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "65d775332f1aec0d79173568ceb89d34"
server_key_iv: "ad29afd1d5ed0cd916357661d5037d2a"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.344353 14916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:17.357654 14916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:17.360175 14916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:17.360304 14916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:17.360486 14916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:17.360803 14916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:17.360838 14916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.360881 14916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:17.360925 14916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.366902 14916 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:43023
I20260926 04:10:17.366976 15044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:43023 every 8 connection(s)
I20260926 04:10:17.367313 14916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:10:17.367882 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 14916
I20260926 04:10:17.367962 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:10:17.368193 855 external_mini_cluster.cc:1428] Setting key 4ffd5f190530c627533d1f42e492b71e
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:17.378085 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.368166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:52307 (local address 127.0.213.254:45691)
0926 04:10:17.368218 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.368220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.368575 (+ 355us) server_negotiation.cc:407] Connection header received
0926 04:10:17.369225 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.369226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.369288 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.369415 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:17.370398 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.370911 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.372047 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.372350 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.372420 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.374874 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:17.374875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:17.374876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:17.374882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:17.376483 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.376913 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.376914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.376914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.376979 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.377386 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.377388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.377388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.377534 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:17.377597 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.377941 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.378014 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:10:17.378511 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.368158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:52307)
0926 04:10:17.368306 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.368483 (+ 177us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.369144 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.369480 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.369484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.370082 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:17.370288 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.370294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.371014 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.371015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.371952 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.371955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.372444 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.372445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.372581 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.373139 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:17.373147 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:17.374768 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:17.376565 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.376571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.376573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.376821 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.377074 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.377075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.377075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.377296 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.377622 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:17.377627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:17.377777 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.377879 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.378038 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260926 04:10:17.379036 15045 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45691
I20260926 04:10:17.379148 15045 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:17.379396 15045 heartbeater.cc:511] Master 127.0.213.254:45691 requested a full tablet report, sending...
I20260926 04:10:17.379743 14586 ts_manager.cc:194] Registered new tserver with Master: d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
I20260926 04:10:17.380218 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:10:17.380321 14586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:52307
I20260926 04:10:17.387823 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.381661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.0.213.254:45691)
0926 04:10:17.381700 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.381702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.381836 (+ 134us) server_negotiation.cc:407] Connection header received
0926 04:10:17.381905 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.381905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.381914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.381963 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:17.382538 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.383102 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.383831 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.384026 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.384174 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.385411 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:17.385413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:17.385413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:17.385419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:17.386664 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.387076 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.387077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.387077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.387140 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.387325 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.387325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.387326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.387485 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:17.387579 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.387657 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.387752 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:10:17.389904 14586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:10:17.391571 14586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:10:17.399443 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.394360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43023 (local address 127.0.0.1:48302)
0926 04:10:17.394537 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.394582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.394651 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.395220 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.395225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.395227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:17.395331 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.395335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.396903 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.396904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.397943 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.397946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.399065 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.399066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.399176 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.399196 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.399282 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.399358 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":16,"thread_start_us":63,"threads_started":1}
I20260926 04:10:17.399467 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.394482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.213.193:34569)
0926 04:10:17.394973 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.394979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.394990 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:10:17.395036 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.395038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.395055 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.395138 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:17.395432 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.396797 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.398087 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.398885 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.399045 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.399121 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.399340 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.399386 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":123,"threads_started":1}
I20260926 04:10:17.399454 15061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.394743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.213.195:43023)
0926 04:10:17.394958 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.394962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.394973 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:10:17.395014 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.395016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.395051 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.395138 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:17.395433 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.396754 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.398071 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.398885 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.399041 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.399120 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.399306 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.399353 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260926 04:10:17.400092 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.394384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:34569 (local address 127.0.0.1:33858)
0926 04:10:17.394702 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.394714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.394777 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.395231 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.395245 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.395247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:17.395331 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.395334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.396897 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.396898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.397991 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.397993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.399062 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.399064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.399176 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.399231 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.399286 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.399984 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":651,"thread_start_us":95,"threads_started":1}
I20260926 04:10:17.400619 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.394592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45791 (local address 127.0.0.1:46248)
0926 04:10:17.395243 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.395253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.395303 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.395516 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.395517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.395519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:17.395587 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.395589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.396634 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.396635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.397990 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.397993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.400373 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.400374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.400446 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.400451 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.400507 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.400550 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260926 04:10:17.400653 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.394715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.0.213.194:45791)
0926 04:10:17.394946 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.394952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.395363 (+ 411us) server_negotiation.cc:407] Connection header received
0926 04:10:17.395417 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.395419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.395433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.395519 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:17.395665 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.396535 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.399163 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.400198 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.400283 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.400339 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.400528 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.400576 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260926 04:10:17.400702 14979 tablet_service.cc:1467] Processing CreateTablet for tablet f3e5687081254ca08ebb4bf17617cb45 (DEFAULT_TABLE table=test-table [id=3b8ef423193744c98708274d9244539c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:17.400830 14711 tablet_service.cc:1467] Processing CreateTablet for tablet f3e5687081254ca08ebb4bf17617cb45 (DEFAULT_TABLE table=test-table [id=3b8ef423193744c98708274d9244539c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:17.401106 14979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e5687081254ca08ebb4bf17617cb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:17.401197 14711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e5687081254ca08ebb4bf17617cb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:17.401707 14845 tablet_service.cc:1467] Processing CreateTablet for tablet f3e5687081254ca08ebb4bf17617cb45 (DEFAULT_TABLE table=test-table [id=3b8ef423193744c98708274d9244539c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:17.402056 14845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e5687081254ca08ebb4bf17617cb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:17.403992 15062 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Bootstrap starting.
I20260926 04:10:17.404632 15062 tablet_bootstrap.cc:654] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:17.404951 15063 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Bootstrap starting.
I20260926 04:10:17.405294 15062 log.cc:824] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:17.405759 15063 tablet_bootstrap.cc:654] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:17.405846 15065 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Bootstrap starting.
I20260926 04:10:17.406405 15063 log.cc:824] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:17.406493 15065 tablet_bootstrap.cc:654] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:17.406545 15062 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: No bootstrap required, opened a new log
I20260926 04:10:17.406657 15062 ts_tablet_manager.cc:1397] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:10:17.407099 15065 log.cc:824] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:17.407240 15063 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: No bootstrap required, opened a new log
I20260926 04:10:17.407310 15063 ts_tablet_manager.cc:1397] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:10:17.407824 15065 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: No bootstrap required, opened a new log
I20260926 04:10:17.407910 15065 ts_tablet_manager.cc:1397] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:10:17.408802 15062 raft_consensus.cc:348] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.408991 15062 raft_consensus.cc:374] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:17.409031 15062 raft_consensus.cc:729] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138fa16b109640ed88f6d901df885755, State: Initialized, Role: FOLLOWER
I20260926 04:10:17.409186 15062 consensus_queue.cc:260] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.409487 15062 ts_tablet_manager.cc:1428] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:10:17.409444 15063 raft_consensus.cc:348] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.409631 14777 heartbeater.cc:503] Master 127.0.213.254:45691 was elected leader, sending a full tablet report...
I20260926 04:10:17.409633 15063 raft_consensus.cc:374] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:17.409693 15063 raft_consensus.cc:729] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d491ccf7c4ee416fb946982438e13f17, State: Initialized, Role: FOLLOWER
I20260926 04:10:17.409807 15063 consensus_queue.cc:260] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.410084 15063 ts_tablet_manager.cc:1428] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:10:17.410133 15045 heartbeater.cc:503] Master 127.0.213.254:45691 was elected leader, sending a full tablet report...
I20260926 04:10:17.410275 15065 raft_consensus.cc:348] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.410459 15065 raft_consensus.cc:374] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:17.410511 15065 raft_consensus.cc:729] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633a007ae0074c9ca56072892b2e6525, State: Initialized, Role: FOLLOWER
I20260926 04:10:17.410638 15065 consensus_queue.cc:260] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.411046 15065 ts_tablet_manager.cc:1428] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:10:17.411192 14911 heartbeater.cc:503] Master 127.0.213.254:45691 was elected leader, sending a full tablet report...
I20260926 04:10:17.425326 15069 raft_consensus.cc:482] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:17.425487 15069 raft_consensus.cc:504] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.425861 15069 leader_election.cc:288] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569), 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791)
I20260926 04:10:17.430344 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.426504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:59163 (local address 127.0.213.194:45791)
0926 04:10:17.426554 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.426556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.426563 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:10:17.426586 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.426587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.426600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.426646 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:17.426811 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.427494 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.429278 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.429902 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.429992 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.430002 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.430220 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.430272 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:10:17.430333 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.426126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:34569 (local address 127.0.213.195:41399)
0926 04:10:17.426193 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.426206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.426264 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.426516 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.426517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.426519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:17.426598 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.426600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.427631 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.427632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.429107 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.429112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.430010 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.430011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.430110 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.430112 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.430183 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.430231 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":87}
I20260926 04:10:17.430342 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.426258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45791 (local address 127.0.213.195:59163)
0926 04:10:17.426431 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.426442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.426486 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.426674 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.426675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.426677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:17.426739 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.426740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.427631 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.427632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.429107 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.429112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.430010 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.430011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.430105 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.430110 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.430228 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.430259 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":92,"thread_start_us":119,"threads_started":1}
I20260926 04:10:17.430399 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.426302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:41399 (local address 127.0.213.193:34569)
0926 04:10:17.426362 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.426364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.426374 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:17.426405 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.426410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.426418 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.426496 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:17.426696 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.427495 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.429287 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.429901 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.429992 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.430002 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.430220 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.430272 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:10:17.430806 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e5687081254ca08ebb4bf17617cb45" candidate_uuid: "d491ccf7c4ee416fb946982438e13f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138fa16b109640ed88f6d901df885755" is_pre_election: true
I20260926 04:10:17.431012 14731 raft_consensus.cc:2454] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d491ccf7c4ee416fb946982438e13f17 in term 0.
I20260926 04:10:17.431020 14865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e5687081254ca08ebb4bf17617cb45" candidate_uuid: "d491ccf7c4ee416fb946982438e13f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633a007ae0074c9ca56072892b2e6525" is_pre_election: true
I20260926 04:10:17.431185 14865 raft_consensus.cc:2454] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d491ccf7c4ee416fb946982438e13f17 in term 0.
I20260926 04:10:17.431264 14935 leader_election.cc:302] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138fa16b109640ed88f6d901df885755, d491ccf7c4ee416fb946982438e13f17; no voters:
I20260926 04:10:17.431411 15069 raft_consensus.cc:2781] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:10:17.431468 15069 raft_consensus.cc:482] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:10:17.431492 15069 raft_consensus.cc:3037] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:17.432291 15069 raft_consensus.cc:504] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.432437 15069 leader_election.cc:288] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [CANDIDATE]: Term 1 election: Requested vote from peers 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569), 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791)
I20260926 04:10:17.432657 14865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e5687081254ca08ebb4bf17617cb45" candidate_uuid: "d491ccf7c4ee416fb946982438e13f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633a007ae0074c9ca56072892b2e6525"
I20260926 04:10:17.432682 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e5687081254ca08ebb4bf17617cb45" candidate_uuid: "d491ccf7c4ee416fb946982438e13f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138fa16b109640ed88f6d901df885755"
I20260926 04:10:17.432787 14731 raft_consensus.cc:3037] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:17.432757 14865 raft_consensus.cc:3037] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:17.433597 14731 raft_consensus.cc:2454] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d491ccf7c4ee416fb946982438e13f17 in term 1.
I20260926 04:10:17.433619 14865 raft_consensus.cc:2454] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d491ccf7c4ee416fb946982438e13f17 in term 1.
I20260926 04:10:17.433784 14935 leader_election.cc:302] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138fa16b109640ed88f6d901df885755, d491ccf7c4ee416fb946982438e13f17; no voters:
I20260926 04:10:17.433908 15069 raft_consensus.cc:2781] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:17.434078 15069 raft_consensus.cc:686] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 1 LEADER]: Becoming Leader. State: Replica: d491ccf7c4ee416fb946982438e13f17, State: Running, Role: LEADER
I20260926 04:10:17.434155 15069 consensus_queue.cc:237] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.435037 14585 catalog_manager.cc:5495] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to d491ccf7c4ee416fb946982438e13f17 (127.0.213.195). New cstate: current_term: 1 leader_uuid: "d491ccf7c4ee416fb946982438e13f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } health_report { overall_health: UNKNOWN } } }
I20260926 04:10:17.445894 15061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.443324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.213.195:43023)
0926 04:10:17.443415 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.443416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.443426 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:17.443460 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.443460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.443478 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.443547 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:17.443693 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.444239 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.445033 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.445266 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.445374 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.445520 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:17.445628 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:10:17.445698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.445779 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.445800 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 04:10:17.448045 14731 raft_consensus.cc:1261] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Refusing update from remote peer d491ccf7c4ee416fb946982438e13f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:17.448107 14865 raft_consensus.cc:1261] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Refusing update from remote peer d491ccf7c4ee416fb946982438e13f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:17.448661 15072 consensus_queue.cc:1035] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:17.448879 15069 consensus_queue.cc:1035] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:17.451285 15075 mvcc.cc:205] Tried to move back new op lower bound from 7333461268264038400 to 7333461268211060736. Current Snapshot: MvccSnapshot[applied={T|T < 7333461268264038400}]
I20260926 04:10:17.454221 14586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:17.454336 14586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:17.455209 14586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Sending DeleteTablet for 3 replicas of tablet f3e5687081254ca08ebb4bf17617cb45
I20260926 04:10:17.455631 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14648
I20260926 04:10:17.455698 14711 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e5687081254ca08ebb4bf17617cb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:33858
I20260926 04:10:17.455719 14979 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e5687081254ca08ebb4bf17617cb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:48302
I20260926 04:10:17.455992 14845 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e5687081254ca08ebb4bf17617cb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:46248
I20260926 04:10:17.458786 15086 tablet_replica.cc:330] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: stopping tablet replica
I20260926 04:10:17.458974 15086 raft_consensus.cc:2229] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:17.459174 15086 raft_consensus.cc:2258] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:17.459877 15086 ts_tablet_manager.cc:1905] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:17.460198 15085 tablet_replica.cc:330] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: stopping tablet replica
I20260926 04:10:17.460362 15085 raft_consensus.cc:2229] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:10:17.460511 15085 raft_consensus.cc:2258] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:17.462298 15086 ts_tablet_manager.cc:1918] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:17.462383 15086 log.cc:1196] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/f3e5687081254ca08ebb4bf17617cb45
I20260926 04:10:17.462751 15086 ts_tablet_manager.cc:1939] T f3e5687081254ca08ebb4bf17617cb45 P 633a007ae0074c9ca56072892b2e6525: Deleting consensus metadata
W20260926 04:10:17.463312 14573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:34569 (error 108)
I20260926 04:10:17.463511 15085 ts_tablet_manager.cc:1905] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:17.463590 14573 catalog_manager.cc:4841] TS 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791): tablet f3e5687081254ca08ebb4bf17617cb45 (table test-table [id=3b8ef423193744c98708274d9244539c]) successfully deleted
W20260926 04:10:17.464112 15059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:10:17.463756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:34569 (local address 127.0.0.1:33868)
0926 04:10:17.463820 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.463833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.463876 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.464027 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:34569: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:10:17.464352 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14782
W20260926 04:10:17.464357 14573 catalog_manager.cc:4584] TS 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3e5687081254ca08ebb4bf17617cb45: Network error: Client connection negotiation failed: client connection to 127.0.213.193:34569: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:10:17.465327 15085 ts_tablet_manager.cc:1918] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:17.465397 15085 log.cc:1196] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/f3e5687081254ca08ebb4bf17617cb45
I20260926 04:10:17.465663 15085 ts_tablet_manager.cc:1939] T f3e5687081254ca08ebb4bf17617cb45 P d491ccf7c4ee416fb946982438e13f17: Deleting consensus metadata
I20260926 04:10:17.466663 14572 catalog_manager.cc:4841] TS d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023): tablet f3e5687081254ca08ebb4bf17617cb45 (table test-table [id=3b8ef423193744c98708274d9244539c]) successfully deleted
I20260926 04:10:17.471210 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14916
I20260926 04:10:17.477228 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 14554
I20260926 04:10:17.483318 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45691
--webserver_interface=127.0.213.254
--webserver_port=46099
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:17.571590 15088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:17.571820 15088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:17.571846 15088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:17.573088 15088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:17.573138 15088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:17.573153 15088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:17.573166 15088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:17.573179 15088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:17.574717 15088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=46099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:17.574986 15088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:17.575242 15088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:17.584964 15095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.584964 15097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.584964 15094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.585009 15088 server_base.cc:1034] running on GCE node
I20260926 04:10:17.585601 15088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:17.586076 15088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:17.587322 15088 hybrid_clock.cc:648] HybridClock initialized: now 1790395817587294 us; error 50 us; skew 500 ppm
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:17.594815 15088 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:17.595422 15088 webserver.cc:460] Webserver started at http://127.0.213.254:46099/ using document root <none> and password file <none>
I20260926 04:10:17.595717 15088 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:17.595816 15088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:17.597653 15088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.598485 15104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.598799 15088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.598901 15088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "36eb6a2c5be644b9b207bcb2227e11bc"
format_stamp: "Formatted at 2026-09-26 04:10:16 on dist-test-slave-xf24"
server_key: "700b79344b4559c3b1feb6e80a637e14"
server_key_iv: "2a7dcd0c63c501f3b0fd0297177ce01b"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.599395 15088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:17.609972 15088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:17.613433 15088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:17.613680 15088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:17.619661 15088 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45691
I20260926 04:10:17.619699 15156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45691 every 8 connection(s)
I20260926 04:10:17.620234 15088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:17.622256 15157 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:10:17.623095 15157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap starting.
I20260926 04:10:17.629304 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15088
I20260926 04:10:17.629634 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:34569
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:45691
--webserver_port=34293
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:17.633716 15157 log.cc:824] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:17.636986 15157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:17.637457 15157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap complete.
I20260926 04:10:17.641912 15157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:17.642324 15157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36eb6a2c5be644b9b207bcb2227e11bc, State: Initialized, Role: FOLLOWER
I20260926 04:10:17.642557 15157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:17.642725 15157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:17.642854 15157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:17.642916 15157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:10:17.644714 15157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:17.644873 15157 leader_election.cc:302] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36eb6a2c5be644b9b207bcb2227e11bc; no voters:
I20260926 04:10:17.645066 15157 leader_election.cc:288] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:10:17.645186 15161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:10:17.645417 15157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:17.645431 15161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 LEADER]: Becoming Leader. State: Replica: 36eb6a2c5be644b9b207bcb2227e11bc, State: Running, Role: LEADER
I20260926 04:10:17.645670 15161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:17.646070 15162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } } }
I20260926 04:10:17.646162 15162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:17.646718 15163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36eb6a2c5be644b9b207bcb2227e11bc. Latest consensus state: current_term: 2 leader_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } } }
I20260926 04:10:17.646803 15163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:17.646973 15170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:17.648017 15170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:17.648190 15170 catalog_manager.cc:1259] Loaded cluster ID: d00dc68c372b4648a172174e478f1c3a
I20260926 04:10:17.648224 15170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:17.648486 15170 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 04:10:17.653668 15170 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 04:10:17.656759 15170 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:17.657048 15170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Loaded TSK: 0
I20260926 04:10:17.657462 15170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:10:17.724944 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:17.725189 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:17.725327 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:17.727031 15159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:17.727104 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:17.727137 15159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:17.728978 15159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:34569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=34293
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:17.729306 15159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:17.729583 15159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:17.735003 15186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.735018 15185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.735028 15188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.735270 15159 server_base.cc:1034] running on GCE node
I20260926 04:10:17.735584 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:17.735869 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:17.737066 15159 hybrid_clock.cc:648] HybridClock initialized: now 1790395817737030 us; error 53 us; skew 500 ppm
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:17.740329 15159 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:17.740859 15159 webserver.cc:460] Webserver started at http://127.0.213.193:34293/ using document root <none> and password file <none>
I20260926 04:10:17.741106 15159 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:17.741204 15159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:17.742724 15159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:10:17.743531 15195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.743826 15159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.743973 15159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "138fa16b109640ed88f6d901df885755"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "0f3178d6eb203cb87068525dadb3a89c"
server_key_iv: "ef9849ecd929ffae4b5c3cdbe63917a9"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.744427 15159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:17.762137 15159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:17.764716 15159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:17.764897 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:17.765147 15159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:17.765628 15202 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:10:17.767776 15159 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:10:17.767886 15159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:10:17.767939 15159 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:10:17.768694 15159 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:10:17.768751 15159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:17.768826 15202 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Bootstrap starting.
I20260926 04:10:17.776221 15159 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:34569
I20260926 04:10:17.776414 15309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:34569 every 8 connection(s)
I20260926 04:10:17.776724 15159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:10:17.779848 15202 log.cc:824] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:17.782919 15202 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:17.783280 15202 tablet_bootstrap.cc:492] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Bootstrap complete.
I20260926 04:10:17.784472 15202 ts_tablet_manager.cc:1397] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260926 04:10:17.785774 15202 raft_consensus.cc:348] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.786037 15202 raft_consensus.cc:729] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138fa16b109640ed88f6d901df885755, State: Initialized, Role: FOLLOWER
I20260926 04:10:17.786216 15202 consensus_queue.cc:260] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } } peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } }
I20260926 04:10:17.786286 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15159
I20260926 04:10:17.786563 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:45791
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:45691
--webserver_port=40589
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:17.786628 15202 ts_tablet_manager.cc:1428] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:17.793792 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.777735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:50161 (local address 127.0.213.254:45691)
0926 04:10:17.780224 (+ 2489us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.780234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.780285 (+ 51us) server_negotiation.cc:407] Connection header received
0926 04:10:17.780440 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.780447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.780697 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.780872 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:17.781977 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.783614 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.785235 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.785956 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.786234 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.790225 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:17.790230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:17.790230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:17.790248 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:17.791956 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.792470 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.792473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.792473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.792566 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.792836 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.792837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.792837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.793071 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:17.793165 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.793460 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.793586 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2427,"thread_start_us":81,"threads_started":1}
I20260926 04:10:17.793799 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.777652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:50161)
0926 04:10:17.777858 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.777905 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.778415 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.780912 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.780916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.781455 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:17.781751 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.781758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.783744 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.783746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.785105 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.785109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.787850 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.787851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.787993 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.788631 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:17.788640 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:17.790033 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:17.792093 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.792112 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.792114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.792362 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.792645 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.792645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.792646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.792759 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.793143 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:17.793150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:17.793294 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.793396 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.793585 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":126,"threads_started":1}
I20260926 04:10:17.794472 15310 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45691
I20260926 04:10:17.794574 15310 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:17.794761 15310 heartbeater.cc:511] Master 127.0.213.254:45691 requested a full tablet report, sending...
I20260926 04:10:17.795532 15121 ts_manager.cc:194] Registered new tserver with Master: 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569)
I20260926 04:10:17.796736 15121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:50161
I20260926 04:10:17.797713 15310 heartbeater.cc:503] Master 127.0.213.254:45691 was elected leader, sending a full tablet report...
I20260926 04:10:17.803896 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.796483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.0.213.193:34569)
0926 04:10:17.800273 (+ 3790us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.800278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.800292 (+ 14us) server_negotiation.cc:407] Connection header received
0926 04:10:17.800359 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.800360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.800374 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.800450 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:17.800609 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.801279 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.802810 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.803476 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.803539 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.803619 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.803782 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.803826 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3750,"thread_start_us":77,"threads_started":1}
I20260926 04:10:17.803894 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.796202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:34569 (local address 127.0.0.1:33882)
0926 04:10:17.800134 (+ 3932us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.800184 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.800266 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.800474 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.800477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.800479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:17.800545 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.800547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.801367 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.801368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.802701 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.802705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.803551 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.803552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.803639 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.803654 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.803744 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.803802 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3900,"thread_start_us":117,"threads_started":1}
I20260926 04:10:17.804756 15244 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e5687081254ca08ebb4bf17617cb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:17 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:33882
I20260926 04:10:17.805321 15319 tablet_replica.cc:330] stopping tablet replica
I20260926 04:10:17.805439 15319 raft_consensus.cc:2229] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:17.805516 15319 raft_consensus.cc:2258] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:17.805847 15319 ts_tablet_manager.cc:1905] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:17.808422 15319 ts_tablet_manager.cc:1918] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:17.808506 15319 log.cc:1196] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/f3e5687081254ca08ebb4bf17617cb45
I20260926 04:10:17.808733 15319 ts_tablet_manager.cc:1939] T f3e5687081254ca08ebb4bf17617cb45 P 138fa16b109640ed88f6d901df885755: Deleting consensus metadata
I20260926 04:10:17.809311 15108 catalog_manager.cc:4841] TS 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569): tablet f3e5687081254ca08ebb4bf17617cb45 (table test-table [id=3b8ef423193744c98708274d9244539c]) successfully deleted
W20260926 04:10:17.875638 15316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:17.875856 15316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:17.875881 15316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:17.877162 15316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:17.877228 15316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:17.877256 15316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:17.878801 15316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:45791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=40589
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:17.879070 15316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:17.879324 15316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:17.884737 15327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.884899 15316 server_base.cc:1034] running on GCE node
W20260926 04:10:17.884739 15325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:17.884730 15324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:17.885227 15316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:17.885454 15316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:17.886605 15316 hybrid_clock.cc:648] HybridClock initialized: now 1790395817886573 us; error 53 us; skew 500 ppm
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:17.889187 15316 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:17.889671 15316 webserver.cc:460] Webserver started at http://127.0.213.194:40589/ using document root <none> and password file <none>
I20260926 04:10:17.889906 15316 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:17.889992 15316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:17.891398 15316 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.892107 15334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.892280 15316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:17.892375 15316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "633a007ae0074c9ca56072892b2e6525"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "28fbfd25dc3346a87c5ec4c1abc9ae58"
server_key_iv: "ea471847c35209163fc99b87fe406c9f"
server_key_version: "encryptionkey@0"
I20260926 04:10:17.892733 15316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:17.928433 15316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:17.930913 15316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:17.931115 15316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:17.931370 15316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:17.931689 15316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:17.931739 15316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.931782 15316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:17.931809 15316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:17.938127 15316 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:45791
I20260926 04:10:17.938169 15447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:45791 every 8 connection(s)
I20260926 04:10:17.938545 15316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:10:17.942858 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15316
I20260926 04:10:17.943195 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:43023
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:45691
--webserver_port=44263
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:10:17 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:17.949945 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.939336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:44347 (local address 127.0.213.254:45691)
0926 04:10:17.939383 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:17.939386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:17.939635 (+ 249us) server_negotiation.cc:407] Connection header received
0926 04:10:17.940153 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:17.940154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:17.940167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:17.940242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:17.941249 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.941761 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.942999 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.943412 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.943480 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.946443 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:17.946446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:17.946446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:17.946456 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:17.948315 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.948875 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.948877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.948877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.948954 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:17.949231 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:17.949232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:17.949232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:17.949366 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:17.949443 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:17.949740 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 04:10:17.949836 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:10:17.950160 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:17.939310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:44347)
0926 04:10:17.939481 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:17.939528 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:17.940014 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:17.940301 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:17.940305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:17.940903 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:17.941108 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.941112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.941862 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.941862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.942844 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:17.942848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:17.943510 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:17.943511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:17.943695 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:17.944393 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:17.944405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:17.946274 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:17.948411 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.948420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.948423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.948796 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.949014 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:17.949015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:17.949015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:17.949144 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:17.949434 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:17.949442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:17.949604 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 04:10:17.949720 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 04:10:17.949898 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260926 04:10:17.950714 15448 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45691
I20260926 04:10:17.950843 15448 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:17.951074 15448 heartbeater.cc:511] Master 127.0.213.254:45691 requested a full tablet report, sending...
I20260926 04:10:17.951479 15121 ts_manager.cc:194] Registered new tserver with Master: 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791)
I20260926 04:10:17.951870 15121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:44347
W20260926 04:10:18.030097 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:18.030333 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:18.030367 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:18.031638 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:18.031687 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:18.031715 15451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:18.033293 15451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:43023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=44263
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:18.033576 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:18.033821 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:18.038995 15457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:18.039057 15451 server_base.cc:1034] running on GCE node
W20260926 04:10:18.039038 15459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:18.039000 15456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:18.039410 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:18.039636 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:18.040773 15451 hybrid_clock.cc:648] HybridClock initialized: now 1790395818040748 us; error 39 us; skew 500 ppm
Sep 26 04:10:18 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:18.043347 15451 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:18.043826 15451 webserver.cc:460] Webserver started at http://127.0.213.195:44263/ using document root <none> and password file <none>
I20260926 04:10:18.044090 15451 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:18.044173 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:18.045706 15451 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:18.046448 15466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:18.046635 15451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:18.046736 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "d491ccf7c4ee416fb946982438e13f17"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "65d775332f1aec0d79173568ceb89d34"
server_key_iv: "ad29afd1d5ed0cd916357661d5037d2a"
server_key_version: "encryptionkey@0"
I20260926 04:10:18.047096 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:18.059049 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:18.061269 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:18.061456 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:18.061699 15451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:18.062043 15451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:18.062098 15451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:18.062139 15451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:18.062165 15451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:18.067999 15451 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:43023
I20260926 04:10:18.068041 15579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:43023 every 8 connection(s)
I20260926 04:10:18.068439 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:10:18.069372 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15451
Sep 26 04:10:18 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:18.077596 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.069753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.0.213.254:45691)
0926 04:10:18.069892 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.069894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.069929 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:10:18.069956 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.069957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.069970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.070034 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:18.070708 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.071467 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.072389 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.072679 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.072758 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.074226 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:18.074228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:18.074228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:18.074235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:18.076224 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:18.076615 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:18.076617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:18.076617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:18.076683 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:18.076927 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:18.076928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:18.076928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:18.077039 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:18.077204 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.077351 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.077470 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 04:10:18.079762 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.069225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:46677 (local address 127.0.213.254:45691)
0926 04:10:18.069360 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.069363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.069485 (+ 122us) server_negotiation.cc:407] Connection header received
0926 04:10:18.070165 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.070166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.070178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.070250 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:18.071352 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.072025 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.073369 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.073665 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.073760 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.076788 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:18.076789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:18.076789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:18.076796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:18.078417 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:18.078829 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:18.078831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:18.078831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:18.078888 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:18.079138 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:18.079139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:18.079140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:18.079246 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:18.079321 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.079596 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.079674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:10:18.079977 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.069197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:46677)
0926 04:10:18.069357 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.069400 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:18.070072 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:18.070307 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:18.070310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:18.070997 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:18.071256 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.071263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.072143 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.072144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.073266 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.073270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.073774 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.073775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.073883 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.074506 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:18.074517 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:18.076700 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:18.078511 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:18.078519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:18.078523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:18.078752 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:18.078952 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:18.078952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:18.078953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:18.079066 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:18.079314 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:18.079321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:18.079465 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 04:10:18.079566 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 04:10:18.079728 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260926 04:10:18.080475 15580 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45691
I20260926 04:10:18.080581 15580 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:18.080763 15580 heartbeater.cc:511] Master 127.0.213.254:45691 requested a full tablet report, sending...
I20260926 04:10:18.081164 15121 ts_manager.cc:194] Registered new tserver with Master: d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
I20260926 04:10:18.081499 15121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:46677
I20260926 04:10:18.089715 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:10:18.092901 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.090134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.0.213.254:45691)
0926 04:10:18.090196 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.090199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.090319 (+ 120us) server_negotiation.cc:407] Connection header received
0926 04:10:18.090443 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.090444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.090490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.090569 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:18.090707 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.091245 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.092046 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.092271 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.092357 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.092555 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:18.092667 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:10:18.092764 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.092783 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.092802 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:10:18.093228 15121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:10:18.094908 15121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:10:18.097584 15244 tablet_service.cc:1467] Processing CreateTablet for tablet 61f6132362c942e9b3cf29875119d68c (DEFAULT_TABLE table=test-table [id=f559e620f0d5472194382ab46dcb6465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:18.097915 15244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f6132362c942e9b3cf29875119d68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:18.099459 15202 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Bootstrap starting.
I20260926 04:10:18.099802 15202 tablet_bootstrap.cc:654] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:18.100358 15202 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: No bootstrap required, opened a new log
I20260926 04:10:18.100421 15202 ts_tablet_manager.cc:1397] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:18.100543 15202 raft_consensus.cc:348] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.100621 15202 raft_consensus.cc:374] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:18.100678 15202 raft_consensus.cc:729] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138fa16b109640ed88f6d901df885755, State: Initialized, Role: FOLLOWER
I20260926 04:10:18.100735 15202 consensus_queue.cc:260] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.100841 15202 ts_tablet_manager.cc:1428] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:18.101413 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.097186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.0.213.194:45791)
0926 04:10:18.097434 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.097438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.097448 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:18.097487 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.097488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.097500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.097594 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:18.097823 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.098749 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.100288 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.101066 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.101138 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.101206 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.101297 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.101347 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":160,"threads_started":1}
I20260926 04:10:18.101421 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.096960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45791 (local address 127.0.0.1:46262)
0926 04:10:18.097012 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.097025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:18.097067 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:18.097731 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:18.097731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:18.097733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:18.097756 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.097757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.098871 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.098872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.100145 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.100147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.101133 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.101134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.101208 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.101212 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:10:18.101284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:10:18.101342 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":43}
I20260926 04:10:18.102267 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.097009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43023 (local address 127.0.0.1:48322)
0926 04:10:18.097170 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.097180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:18.097234 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:18.097588 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:18.097590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:18.097592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:18.097648 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.097650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.098927 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.098928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.100144 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.100147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.102006 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.102007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.102074 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.102079 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:18.102132 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:10:18.102178 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":43,"thread_start_us":102,"threads_started":1}
I20260926 04:10:18.102226 15382 tablet_service.cc:1467] Processing CreateTablet for tablet 61f6132362c942e9b3cf29875119d68c (DEFAULT_TABLE table=test-table [id=f559e620f0d5472194382ab46dcb6465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:18.102504 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.097237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.213.195:43023)
0926 04:10:18.097418 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.097423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.097435 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:10:18.097478 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.097481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.097497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.097579 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:18.097755 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.098835 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.100906 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.101868 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.101940 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.102011 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.102373 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.102422 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260926 04:10:18.102612 15382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f6132362c942e9b3cf29875119d68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:18.105578 15590 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Bootstrap starting.
I20260926 04:10:18.106108 15514 tablet_service.cc:1467] Processing CreateTablet for tablet 61f6132362c942e9b3cf29875119d68c (DEFAULT_TABLE table=test-table [id=f559e620f0d5472194382ab46dcb6465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:18.106199 15590 tablet_bootstrap.cc:654] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:18.106419 15514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61f6132362c942e9b3cf29875119d68c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:18.106766 15590 log.cc:824] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:18.107687 15590 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: No bootstrap required, opened a new log
I20260926 04:10:18.107775 15590 ts_tablet_manager.cc:1397] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:10:18.109236 15590 raft_consensus.cc:348] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.109563 15590 raft_consensus.cc:374] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:18.109552 15592 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Bootstrap starting.
I20260926 04:10:18.109625 15590 raft_consensus.cc:729] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633a007ae0074c9ca56072892b2e6525, State: Initialized, Role: FOLLOWER
I20260926 04:10:18.109772 15590 consensus_queue.cc:260] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.110116 15590 ts_tablet_manager.cc:1428] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:10:18.110138 15448 heartbeater.cc:503] Master 127.0.213.254:45691 was elected leader, sending a full tablet report...
I20260926 04:10:18.110378 15592 tablet_bootstrap.cc:654] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:18.111040 15592 log.cc:824] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:18.111894 15592 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: No bootstrap required, opened a new log
I20260926 04:10:18.111958 15592 ts_tablet_manager.cc:1397] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:10:18.113487 15592 raft_consensus.cc:348] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.113655 15592 raft_consensus.cc:374] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:18.113682 15592 raft_consensus.cc:729] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d491ccf7c4ee416fb946982438e13f17, State: Initialized, Role: FOLLOWER
I20260926 04:10:18.113821 15592 consensus_queue.cc:260] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.114159 15592 ts_tablet_manager.cc:1428] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:10:18.114179 15580 heartbeater.cc:503] Master 127.0.213.254:45691 was elected leader, sending a full tablet report...
W20260926 04:10:18.189167 15449 tablet.cc:2369] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:10:18.281078 15311 tablet.cc:2369] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:10:18.319068 15581 tablet.cc:2369] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:10:18.563387 15315 raft_consensus.cc:482] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:18.563549 15315 raft_consensus.cc:504] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.563959 15315 leader_election.cc:288] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791), d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
I20260926 04:10:18.566897 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.564354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:51841 (local address 127.0.213.194:45791)
0926 04:10:18.564445 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.564447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.564458 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:10:18.564493 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.564494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.564504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.564559 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:18.564715 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.565209 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.566180 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.566559 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.566643 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.566651 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.566779 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.566821 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 04:10:18.566905 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.564292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:58525 (local address 127.0.213.195:43023)
0926 04:10:18.564343 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.564346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.564594 (+ 248us) server_negotiation.cc:407] Connection header received
0926 04:10:18.564616 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.564617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.564623 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.564682 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:18.564834 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.565296 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.566179 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.566550 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.566632 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.566641 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.566796 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.566835 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:10:18.566927 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.564213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45791 (local address 127.0.213.193:51841)
0926 04:10:18.564361 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.564372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:18.564432 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:18.564574 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:18.564575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:18.564578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:18.564630 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.564632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.565294 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.565295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.566083 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.566086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.566647 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.566648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.566708 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.566712 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:10:18.566775 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:10:18.566821 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"spinlock_wait_cycles":15744,"thread_start_us":83,"threads_started":1}
I20260926 04:10:18.566941 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.564211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43023 (local address 127.0.213.193:58525)
0926 04:10:18.564349 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.564361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:18.564445 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:18.564687 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:18.564688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:18.564690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:18.564752 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.564754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.565376 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.565377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.566087 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:18.566089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.566644 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.566645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:18.566710 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.566719 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 04:10:18.566778 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 04:10:18.566835 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260926 04:10:18.567310 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f6132362c942e9b3cf29875119d68c" candidate_uuid: "138fa16b109640ed88f6d901df885755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633a007ae0074c9ca56072892b2e6525" is_pre_election: true
I20260926 04:10:18.567294 15534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f6132362c942e9b3cf29875119d68c" candidate_uuid: "138fa16b109640ed88f6d901df885755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d491ccf7c4ee416fb946982438e13f17" is_pre_election: true
I20260926 04:10:18.567492 15402 raft_consensus.cc:2454] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138fa16b109640ed88f6d901df885755 in term 0.
I20260926 04:10:18.567500 15534 raft_consensus.cc:2454] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138fa16b109640ed88f6d901df885755 in term 0.
I20260926 04:10:18.567731 15199 leader_election.cc:302] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138fa16b109640ed88f6d901df885755, 633a007ae0074c9ca56072892b2e6525; no voters:
I20260926 04:10:18.567886 15315 raft_consensus.cc:2781] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:10:18.567942 15315 raft_consensus.cc:482] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:10:18.567960 15315 raft_consensus.cc:3037] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:18.568749 15315 raft_consensus.cc:504] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.568902 15315 leader_election.cc:288] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 1 election: Requested vote from peers 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791), d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
I20260926 04:10:18.569083 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f6132362c942e9b3cf29875119d68c" candidate_uuid: "138fa16b109640ed88f6d901df885755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633a007ae0074c9ca56072892b2e6525"
I20260926 04:10:18.569089 15534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61f6132362c942e9b3cf29875119d68c" candidate_uuid: "138fa16b109640ed88f6d901df885755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d491ccf7c4ee416fb946982438e13f17"
I20260926 04:10:18.569152 15402 raft_consensus.cc:3037] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:18.569190 15534 raft_consensus.cc:3037] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:18.569957 15534 raft_consensus.cc:2454] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138fa16b109640ed88f6d901df885755 in term 1.
I20260926 04:10:18.569968 15402 raft_consensus.cc:2454] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138fa16b109640ed88f6d901df885755 in term 1.
I20260926 04:10:18.570165 15198 leader_election.cc:302] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138fa16b109640ed88f6d901df885755, d491ccf7c4ee416fb946982438e13f17; no voters:
I20260926 04:10:18.570259 15315 raft_consensus.cc:2781] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:18.570394 15315 raft_consensus.cc:686] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 LEADER]: Becoming Leader. State: Replica: 138fa16b109640ed88f6d901df885755, State: Running, Role: LEADER
I20260926 04:10:18.570477 15315 consensus_queue.cc:237] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.571274 15119 catalog_manager.cc:5495] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 138fa16b109640ed88f6d901df885755 (127.0.213.193). New cstate: current_term: 1 leader_uuid: "138fa16b109640ed88f6d901df885755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } health_report { overall_health: HEALTHY } } }
I20260926 04:10:18.654886 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.652305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.0.213.193:34569)
0926 04:10:18.652419 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:18.652422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:18.652435 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:10:18.652483 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:18.652483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:18.652503 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:18.652581 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:18.652700 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.653231 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.654059 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:18.654270 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:18.654336 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:18.654541 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:18.654659 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:10:18.654729 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:18.654776 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 04:10:18.654798 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":106,"threads_started":1}
I20260926 04:10:18.656440 15402 raft_consensus.cc:1261] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Refusing update from remote peer 138fa16b109640ed88f6d901df885755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:18.656533 15534 raft_consensus.cc:1261] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Refusing update from remote peer 138fa16b109640ed88f6d901df885755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:18.657085 15598 consensus_queue.cc:1035] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:18.657238 15315 consensus_queue.cc:1035] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:18.660877 15607 mvcc.cc:205] Tried to move back new op lower bound from 7333461273214083072 to 7333461272865140736. Current Snapshot: MvccSnapshot[applied={T|T < 7333461273214083072}]
I20260926 04:10:18.662084 15119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:18.662173 15119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:18.662982 15119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Sending DeleteTablet for 3 replicas of tablet 61f6132362c942e9b3cf29875119d68c
I20260926 04:10:18.663316 15382 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f6132362c942e9b3cf29875119d68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:18 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:46262
I20260926 04:10:18.663339 15514 tablet_service.cc:1514] Processing DeleteTablet for tablet 61f6132362c942e9b3cf29875119d68c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:18 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:48322
I20260926 04:10:18.663573 15615 tablet_replica.cc:330] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: stopping tablet replica
I20260926 04:10:18.663604 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15159
I20260926 04:10:18.663699 15615 raft_consensus.cc:2229] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:18.663805 15615 raft_consensus.cc:2258] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:18.664283 15615 ts_tablet_manager.cc:1905] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:18.666431 15615 ts_tablet_manager.cc:1918] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:18.666510 15615 log.cc:1196] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/61f6132362c942e9b3cf29875119d68c
I20260926 04:10:18.666778 15615 ts_tablet_manager.cc:1939] T 61f6132362c942e9b3cf29875119d68c P 633a007ae0074c9ca56072892b2e6525: Deleting consensus metadata
I20260926 04:10:18.667603 15108 catalog_manager.cc:4841] TS 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791): tablet 61f6132362c942e9b3cf29875119d68c (table test-table [id=f559e620f0d5472194382ab46dcb6465]) successfully deleted
I20260926 04:10:18.667866 15616 tablet_replica.cc:330] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: stopping tablet replica
I20260926 04:10:18.667970 15616 raft_consensus.cc:2229] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:18.668111 15616 raft_consensus.cc:2258] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:18.668485 15616 ts_tablet_manager.cc:1905] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:18.669778 15616 ts_tablet_manager.cc:1918] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:18.669844 15616 log.cc:1196] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/61f6132362c942e9b3cf29875119d68c
I20260926 04:10:18.670100 15616 ts_tablet_manager.cc:1939] T 61f6132362c942e9b3cf29875119d68c P d491ccf7c4ee416fb946982438e13f17: Deleting consensus metadata
I20260926 04:10:18.670776 15107 catalog_manager.cc:4841] TS d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023): tablet 61f6132362c942e9b3cf29875119d68c (table test-table [id=f559e620f0d5472194382ab46dcb6465]) successfully deleted
W20260926 04:10:18.671092 15108 connection.cc:535] client connection to 127.0.213.193:34569 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 04:10:18.671198 15108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:10:18.671380 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15316
I20260926 04:10:18.673517 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:18.673322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:34569 (local address 127.0.0.1:33900)
0926 04:10:18.673433 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.673481 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:34569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
W20260926 04:10:18.673643 15108 catalog_manager.cc:4584] TS 138fa16b109640ed88f6d901df885755 (127.0.213.193:34569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61f6132362c942e9b3cf29875119d68c: Network error: Client connection negotiation failed: client connection to 127.0.213.193:34569: connect: Connection refused (error 111)
I20260926 04:10:18.676000 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15451
I20260926 04:10:18.681018 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15088
I20260926 04:10:18.686335 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45691
--webserver_interface=127.0.213.254
--webserver_port=46099
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:18.769047 15619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:18.769308 15619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:18.769348 15619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:18.770551 15619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:18.770620 15619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:18.770645 15619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:18.770673 15619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:18.770701 15619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:18.772269 15619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=46099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:18.772559 15619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:18.772794 15619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:18.777797 15624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:18.777808 15627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:18.777808 15625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:18.777981 15619 server_base.cc:1034] running on GCE node
I20260926 04:10:18.778251 15619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:18.778529 15619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:18.779678 15619 hybrid_clock.cc:648] HybridClock initialized: now 1790395818779665 us; error 31 us; skew 500 ppm
Sep 26 04:10:18 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:18.782471 15619 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:18.782958 15619 webserver.cc:460] Webserver started at http://127.0.213.254:46099/ using document root <none> and password file <none>
I20260926 04:10:18.783159 15619 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:18.783201 15619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:18.784638 15619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:18.785348 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:18.785553 15619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:18.785643 15619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "36eb6a2c5be644b9b207bcb2227e11bc"
format_stamp: "Formatted at 2026-09-26 04:10:16 on dist-test-slave-xf24"
server_key: "700b79344b4559c3b1feb6e80a637e14"
server_key_iv: "2a7dcd0c63c501f3b0fd0297177ce01b"
server_key_version: "encryptionkey@0"
I20260926 04:10:18.786005 15619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:18.806514 15619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:18.809013 15619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:18.809135 15619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:18.813150 15619 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45691
I20260926 04:10:18.813221 15686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45691 every 8 connection(s)
I20260926 04:10:18.813598 15619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:18.815241 15687 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:10:18.815855 15687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap starting.
I20260926 04:10:18.821661 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15619
I20260926 04:10:18.821973 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:34569
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:45691
--webserver_port=34293
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:18.824836 15687 log.cc:824] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:18.827423 15687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:18.827785 15687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc: Bootstrap complete.
I20260926 04:10:18.830190 15687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:18.830461 15687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36eb6a2c5be644b9b207bcb2227e11bc, State: Initialized, Role: FOLLOWER
I20260926 04:10:18.830646 15687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:18.830722 15687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:18.830778 15687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:18.830828 15687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 2 FOLLOWER]: Advancing to term 3
I20260926 04:10:18.831867 15687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:18.831988 15687 leader_election.cc:302] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36eb6a2c5be644b9b207bcb2227e11bc; no voters:
I20260926 04:10:18.832221 15687 leader_election.cc:288] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 04:10:18.832260 15691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 3 FOLLOWER]: Leader election won for term 3
I20260926 04:10:18.832383 15691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [term 3 LEADER]: Becoming Leader. State: Replica: 36eb6a2c5be644b9b207bcb2227e11bc, State: Running, Role: LEADER
I20260926 04:10:18.832471 15691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } }
I20260926 04:10:18.832504 15687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:18.834226 15692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } } }
I20260926 04:10:18.834321 15692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:18.834373 15693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36eb6a2c5be644b9b207bcb2227e11bc. Latest consensus state: current_term: 3 leader_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36eb6a2c5be644b9b207bcb2227e11bc" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45691 } } }
I20260926 04:10:18.834473 15693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36eb6a2c5be644b9b207bcb2227e11bc [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:18.834525 15707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:18.835515 15707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:18.835640 15707 catalog_manager.cc:1259] Loaded cluster ID: d00dc68c372b4648a172174e478f1c3a
I20260926 04:10:18.835675 15707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 04:10:18.835906 15707 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9fea2469ad google::LogMessage::Fail() at ??:0
@ 0x7f9fea24988a google::LogMessage::SendToLog() at ??:0
@ 0x7f9fea2464de google::LogMessage::Flush() at ??:0
@ 0x7f9fea247a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9ff216b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9ff21748b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9fec1140d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9fec109954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9fecf6d6db start_thread at ??:0
@ 0x7f9fe8d4461f clone at ??:0
W20260926 04:10:18.919627 15689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:18.919927 15689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:18.919973 15689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:18.921388 15689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:18.921476 15689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:18.921528 15689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:18.923182 15689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:34569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=34293
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:18.923482 15689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:18.923730 15689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:10:18.931039 15689 server_base.cc:1034] running on GCE node
W20260926 04:10:18.930799 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:18.930960 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:18.930881 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:18.931535 15689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:18.931861 15689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:18.933028 15689 hybrid_clock.cc:648] HybridClock initialized: now 1790395818933014 us; error 33 us; skew 500 ppm
Sep 26 04:10:18 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:18.936025 15689 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:18.936553 15689 webserver.cc:460] Webserver started at http://127.0.213.193:34293/ using document root <none> and password file <none>
I20260926 04:10:18.936797 15689 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:18.936844 15689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:18.938460 15689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:10:18.939247 15733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:18.939429 15689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:18.939553 15689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "138fa16b109640ed88f6d901df885755"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "0f3178d6eb203cb87068525dadb3a89c"
server_key_iv: "ef9849ecd929ffae4b5c3cdbe63917a9"
server_key_version: "encryptionkey@0"
I20260926 04:10:18.939952 15689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:18.956579 15689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:18.959436 15689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:18.959631 15689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:18.959885 15689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:18.960439 15740 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:10:18.962474 15689 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:10:18.962564 15689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:10:18.962617 15689 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:10:18.963377 15689 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:10:18.963435 15689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:18.963505 15740 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Bootstrap starting.
I20260926 04:10:18.970820 15689 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:34569
I20260926 04:10:18.970984 15847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:34569 every 8 connection(s)
I20260926 04:10:18.971271 15689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:10:18.974677 15740 log.cc:824] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:18.977738 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15689
I20260926 04:10:18.977751 15740 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:18.978103 15740 tablet_bootstrap.cc:492] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Bootstrap complete.
I20260926 04:10:18.978139 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:45791
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:45691
--webserver_port=40589
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:18.979394 15740 ts_tablet_manager.cc:1397] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260926 04:10:18.981076 15740 raft_consensus.cc:348] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.981454 15740 raft_consensus.cc:729] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138fa16b109640ed88f6d901df885755, State: Initialized, Role: FOLLOWER
I20260926 04:10:18.981628 15740 consensus_queue.cc:260] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:18.981925 15740 ts_tablet_manager.cc:1428] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260926 04:10:19.050957 15850 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:10:18.971978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:44307)
0926 04:10:18.972178 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:18.972229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:18.972700 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:19.050734 (+ 78034us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":127,"threads_started":1}
W20260926 04:10:19.051396 15848 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45691 (0 consecutive failures): Network error: Failed to ping master at 127.0.213.254:45691: Client connection negotiation failed: client connection to 127.0.213.254:45691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:10:19.051774 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.051635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:55661)
0926 04:10:19.051687 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.051738 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 04:10:19.052191 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.052083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:60779)
0926 04:10:19.052134 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.052170 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 04:10:19.052335 15848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:10:19.067641 15852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:19.067866 15852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:19.067893 15852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:19.069216 15852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:19.069290 15852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:19.069321 15852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:19.070930 15852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:45791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=40589
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:19.071206 15852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:19.071492 15852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:19.076994 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:19.077121 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:19.077065 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:19.077215 15852 server_base.cc:1034] running on GCE node
I20260926 04:10:19.077433 15852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:19.077679 15852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:19.078830 15852 hybrid_clock.cc:648] HybridClock initialized: now 1790395819078806 us; error 57 us; skew 500 ppm
Sep 26 04:10:19 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:19.081629 15852 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:19.082120 15852 webserver.cc:460] Webserver started at http://127.0.213.194:40589/ using document root <none> and password file <none>
I20260926 04:10:19.082358 15852 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:19.082425 15852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:19.083927 15852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:19.084697 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:19.084954 15852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:19.085064 15852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "633a007ae0074c9ca56072892b2e6525"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "28fbfd25dc3346a87c5ec4c1abc9ae58"
server_key_iv: "ea471847c35209163fc99b87fe406c9f"
server_key_version: "encryptionkey@0"
I20260926 04:10:19.085476 15852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:19.099879 15852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:19.102415 15852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:19.102586 15852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:19.102831 15852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:19.103142 15852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:19.103199 15852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:19.103255 15852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:19.103300 15852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:19.109942 15852 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:45791
I20260926 04:10:19.110025 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:45791 every 8 connection(s)
I20260926 04:10:19.110392 15852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:10:19.111770 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.111111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:47467)
0926 04:10:19.111307 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.111409 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
W20260926 04:10:19.112098 15982 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45691 (0 consecutive failures): Network error: Failed to ping master at 127.0.213.254:45691: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
I20260926 04:10:19.112447 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.112301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:34561)
0926 04:10:19.112349 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.112425 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 04:10:19.112783 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.112703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:48595)
0926 04:10:19.112745 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.112768 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:10:19.112937 15982 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:10:19.114533 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15852
I20260926 04:10:19.114774 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:43023
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:45691
--webserver_port=44263
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:19.201301 15985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:19.201532 15985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:19.201560 15985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:19.202757 15985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:19.202795 15985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:19.202822 15985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:19.204584 15985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:43023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=44263
--tserver_master_addrs=127.0.213.254:45691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:19.204821 15985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:19.205013 15985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:19.210239 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:19.210235 15990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:19.210239 15991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:19.210494 15985 server_base.cc:1034] running on GCE node
I20260926 04:10:19.210757 15985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:19.210981 15985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:19.212146 15985 hybrid_clock.cc:648] HybridClock initialized: now 1790395819212130 us; error 45 us; skew 500 ppm
Sep 26 04:10:19 dist-test-slave-xf24 krb5kdc[14541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:19.215003 15985 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:19.215569 15985 webserver.cc:460] Webserver started at http://127.0.213.195:44263/ using document root <none> and password file <none>
I20260926 04:10:19.215829 15985 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:19.215898 15985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:19.217443 15985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:19.218143 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:19.218397 15985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:19.218523 15985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "d491ccf7c4ee416fb946982438e13f17"
format_stamp: "Formatted at 2026-09-26 04:10:17 on dist-test-slave-xf24"
server_key: "65d775332f1aec0d79173568ceb89d34"
server_key_iv: "ad29afd1d5ed0cd916357661d5037d2a"
server_key_version: "encryptionkey@0"
I20260926 04:10:19.218940 15985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:19.236536 15985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:19.239102 15985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:19.239289 15985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:19.239548 15985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:19.239874 15985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:19.239926 15985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:19.239969 15985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:19.240017 15985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:19.247167 15985 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:43023
I20260926 04:10:19.247221 16113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:43023 every 8 connection(s)
I20260926 04:10:19.247620 15985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:10:19.248821 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.248396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:39409)
0926 04:10:19.248581 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.248658 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
W20260926 04:10:19.249301 16114 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45691 (0 consecutive failures): Network error: Failed to ping master at 127.0.213.254:45691: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
I20260926 04:10:19.249733 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.249587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:53219)
0926 04:10:19.249665 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.249709 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 04:10:19.250209 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:19.250049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:59253)
0926 04:10:19.250107 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:19.250178 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 04:10:19.250360 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 15985
W20260926 04:10:19.250541 16114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:10:20.052966 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.052736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:43735)
0926 04:10:20.052858 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:20.052902 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260926 04:10:20.113502 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.113278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:54517)
0926 04:10:20.113413 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:20.113448 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260926 04:10:20.197295 16120 raft_consensus.cc:482] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:20.197485 16120 raft_consensus.cc:504] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:20.197813 16120 leader_election.cc:288] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791), d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
Sep 26 04:10:20 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.195@KRBTEST.COM
Sep 26 04:10:20 dist-test-slave-xf24 krb5kdc[14541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.194@KRBTEST.COM
I20260926 04:10:20.215683 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.198041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43023 (local address 127.0.213.193:60899)
0926 04:10:20.198165 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:20.198175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:20.198219 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:20.199064 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:20.199065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:20.199384 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:20.199579 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:20.199585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.200962 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.200963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:20.201746 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:20.201748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.202408 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.202410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:20.202502 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:20.208935 (+ 6433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:20.208947 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:20.210730 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:20.213243 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:20.213251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:20.213253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:20.213789 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:20.214145 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:20.214146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:20.214147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:20.214334 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:20.214983 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:20.214991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:20.215172 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 04:10:20.215307 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 04:10:20.215537 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260926 04:10:20.215704 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.198128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:60899 (local address 127.0.213.195:43023)
0926 04:10:20.198335 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:20.198344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:20.198364 (+ 20us) server_negotiation.cc:407] Connection header received
0926 04:10:20.198824 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:20.198826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:20.198924 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:20.199069 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:20.199710 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.200814 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.201860 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.202294 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.202437 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:20.210921 (+ 8484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:20.210927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:20.210927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:20.210940 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:20.213118 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:20.213915 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:20.213918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:20.213919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:20.214058 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:20.214440 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:20.214442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:20.214442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:20.214904 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:20.214992 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:20.215417 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 04:10:20.215584 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":52,"threads_started":1}
I20260926 04:10:20.215688 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.198021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:45791 (local address 127.0.213.193:35311)
0926 04:10:20.198109 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:20.198123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:20.198186 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:20.198990 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:20.198993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:20.199386 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:20.199579 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:20.199585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.200793 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.200794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:20.201748 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:20.201751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.202375 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.202376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:20.202500 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:20.208945 (+ 6445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:20.208950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:20.211048 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:20.213340 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:20.213341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:20.213341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:20.213789 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:20.214145 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:20.214146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:20.214147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:20.214334 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:20.214990 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:20.214991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:20.215172 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 04:10:20.215307 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 04:10:20.215595 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":179}
I20260926 04:10:20.215718 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.198128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:35311 (local address 127.0.213.194:45791)
0926 04:10:20.198284 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:20.198294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:20.198322 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:10:20.198742 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:20.198744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:20.198784 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:20.198895 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:20.199728 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.200691 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.201867 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:20.202277 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:20.202397 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:20.211174 (+ 8777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:20.211179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:20.211179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:20.211193 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:20.213258 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:20.213914 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:20.213918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:20.213919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:20.214058 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:20.214440 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:20.214442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:20.214442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:20.214870 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:20.214978 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:20.215458 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 04:10:20.215595 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":55,"threads_started":1}
W20260926 04:10:20.216758 15736 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
W20260926 04:10:20.217172 15737 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
I20260926 04:10:20.217273 15737 leader_election.cc:302] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138fa16b109640ed88f6d901df885755; no voters: 633a007ae0074c9ca56072892b2e6525, d491ccf7c4ee416fb946982438e13f17
I20260926 04:10:20.217409 16120 raft_consensus.cc:2726] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:10:20.251128 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:20.250884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:51179)
0926 04:10:20.251037 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:20.251078 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260926 04:10:21.053841 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:21.053576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:37305)
0926 04:10:21.053746 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:21.053794 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260926 04:10:21.114267 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:21.114031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:37709)
0926 04:10:21.114174 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:21.114222 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260926 04:10:21.251891 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:21.251626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:56517)
0926 04:10:21.251764 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:21.251806 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260926 04:10:22.049770 16128 raft_consensus.cc:482] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:22.049875 16128 raft_consensus.cc:504] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:22.050069 16128 leader_election.cc:288] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791), d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
W20260926 04:10:22.050597 15737 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
W20260926 04:10:22.050714 15736 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
I20260926 04:10:22.050765 15736 leader_election.cc:302] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138fa16b109640ed88f6d901df885755; no voters: 633a007ae0074c9ca56072892b2e6525, d491ccf7c4ee416fb946982438e13f17
I20260926 04:10:22.050868 16128 raft_consensus.cc:2726] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:10:22.054593 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:22.054386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:44091)
0926 04:10:22.054510 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:22.054549 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260926 04:10:22.115077 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:22.114799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:45761)
0926 04:10:22.114975 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:22.115030 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260926 04:10:22.252740 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:22.252475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:58637)
0926 04:10:22.252647 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:22.252693 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260926 04:10:23.055415 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:23.055145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:54953)
0926 04:10:23.055321 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:23.055369 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260926 04:10:23.116132 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:23.115781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:52331)
0926 04:10:23.115961 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:23.116019 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260926 04:10:23.253715 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:23.253330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:39461)
0926 04:10:23.253617 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:23.253664 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
I20260926 04:10:24.056494 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:24.056248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:49117)
0926 04:10:24.056388 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:24.056438 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260926 04:10:24.117125 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:24.116793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:38133)
0926 04:10:24.117025 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:24.117076 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260926 04:10:24.254645 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:24.254387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:35427)
0926 04:10:24.254546 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:24.254595 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260926 04:10:24.616916 16138 raft_consensus.cc:482] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:24.617040 16138 raft_consensus.cc:504] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:24.617234 16138 leader_election.cc:288] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791), d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
W20260926 04:10:24.617728 15736 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
W20260926 04:10:24.617807 15737 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
I20260926 04:10:24.617857 15737 leader_election.cc:302] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138fa16b109640ed88f6d901df885755; no voters: 633a007ae0074c9ca56072892b2e6525, d491ccf7c4ee416fb946982438e13f17
I20260926 04:10:24.617947 16138 raft_consensus.cc:2726] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:10:25.057340 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:25.057063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:58813)
0926 04:10:25.057242 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:25.057291 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260926 04:10:25.118042 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:25.117793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:54203)
0926 04:10:25.117944 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:25.117988 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260926 04:10:25.255543 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:25.255275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:52195)
0926 04:10:25.255445 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:25.255495 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260926 04:10:26.058280 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:26.058014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:59797)
0926 04:10:26.058183 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:26.058230 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260926 04:10:26.118822 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:26.118568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:55295)
0926 04:10:26.118740 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:26.118776 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260926 04:10:26.256618 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:26.256266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:39887)
0926 04:10:26.256516 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:26.256564 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":138,"threads_started":1}
I20260926 04:10:27.059218 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:27.058923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:44737)
0926 04:10:27.059070 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:27.059116 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260926 04:10:27.119737 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:27.119473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:51493)
0926 04:10:27.119643 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:27.119692 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260926 04:10:27.257622 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:27.257278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:56329)
0926 04:10:27.257466 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:27.257527 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260926 04:10:27.975051 16148 raft_consensus.cc:482] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:27.975241 16148 raft_consensus.cc:504] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633a007ae0074c9ca56072892b2e6525" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 45791 } } peers { permanent_uuid: "d491ccf7c4ee416fb946982438e13f17" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43023 } } peers { permanent_uuid: "138fa16b109640ed88f6d901df885755" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 34569 } }
I20260926 04:10:27.975466 16148 leader_election.cc:288] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791), d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023)
W20260926 04:10:27.975972 15737 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633a007ae0074c9ca56072892b2e6525 (127.0.213.194:45791): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
W20260926 04:10:27.976034 15736 leader_election.cc:341] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d491ccf7c4ee416fb946982438e13f17 (127.0.213.195:43023): Not found: Tablet not found: 61f6132362c942e9b3cf29875119d68c
I20260926 04:10:27.976089 15737 leader_election.cc:302] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 138fa16b109640ed88f6d901df885755; no voters: 633a007ae0074c9ca56072892b2e6525, d491ccf7c4ee416fb946982438e13f17
I20260926 04:10:27.976172 16148 raft_consensus.cc:2726] T 61f6132362c942e9b3cf29875119d68c P 138fa16b109640ed88f6d901df885755 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:10:28.060267 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:28.059885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:39693)
0926 04:10:28.060151 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:28.060216 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":72,"threads_started":1}
I20260926 04:10:28.120723 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:28.120468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:33497)
0926 04:10:28.120609 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:28.120657 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":122,"threads_started":1}
I20260926 04:10:28.258613 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:28.258337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:55261)
0926 04:10:28.258513 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:28.258563 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260926 04:10:29.061312 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:29.061025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:50609)
0926 04:10:29.061208 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:29.061260 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260926 04:10:29.121810 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:29.121486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:45843)
0926 04:10:29.121712 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:29.121759 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260926 04:10:29.259570 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:29.259297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:56381)
0926 04:10:29.259472 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:29.259520 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260926 04:10:30.062142 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:30.061896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:39033)
0926 04:10:30.062042 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:30.062089 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260926 04:10:30.122735 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:30.122488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:40967)
0926 04:10:30.122635 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:30.122673 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260926 04:10:30.260552 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:30.260254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:45355)
0926 04:10:30.260431 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:30.260468 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260926 04:10:31.063081 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:31.062828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:49821)
0926 04:10:31.062982 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:31.063019 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260926 04:10:31.123682 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:31.123372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:48963)
0926 04:10:31.123580 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:31.123626 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260926 04:10:31.261529 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:31.261242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:44943)
0926 04:10:31.261423 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:31.261478 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260926 04:10:32.064219 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:32.063840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:37275)
0926 04:10:32.064105 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:32.064159 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260926 04:10:32.124600 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:32.124308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:44857)
0926 04:10:32.124507 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:32.124553 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260926 04:10:32.262461 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:32.262204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:38239)
0926 04:10:32.262363 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:32.262411 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260926 04:10:33.065060 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:33.064765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:50897)
0926 04:10:33.064953 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:33.065011 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260926 04:10:33.125504 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:33.125251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:48005)
0926 04:10:33.125411 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:33.125454 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260926 04:10:33.263312 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:33.263070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:54457)
0926 04:10:33.263222 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:33.263261 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260926 04:10:34.065948 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:34.065667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.193:60207)
0926 04:10:34.065863 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:34.065900 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20260926 04:10:34.126451 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:34.126203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.194:57449)
0926 04:10:34.126341 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:34.126390 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260926 04:10:34.257606 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15689
I20260926 04:10:34.262490 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15852
I20260926 04:10:34.264449 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:34.264233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45691 (local address 127.0.213.195:53939)
0926 04:10:34.264366 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:34.264406 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260926 04:10:34.267123 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 15985
2026-09-26T04:10:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18740 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:10:34 dist-test-slave-xf24 krb5kdc[16172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:10:34 dist-test-slave-xf24 krb5kdc[16172](info): set up 2 sockets
Sep 26 04:10:34 dist-test-slave-xf24 krb5kdc[16172](info): commencing operation
krb5kdc: starting...
W20260926 04:10:35.348548 855 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:10:35 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:10:35.484334 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45025
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:35.567123 16185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:35.567397 16185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:35.567441 16185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:35.568672 16185 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 04:10:35.568749 16185 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 04:10:35.568771 16185 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 04:10:35.568790 16185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:35.568823 16185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:35.568846 16185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:35.568871 16185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:35.568898 16185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:35.570426 16185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:35.570719 16185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:35.570962 16185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:35.575970 16193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:35.575978 16191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:35.575978 16190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:35.576316 16185 server_base.cc:1034] running on GCE node
I20260926 04:10:35.576615 16185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:35.576944 16185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:35.578117 16185 hybrid_clock.cc:648] HybridClock initialized: now 1790395835578094 us; error 42 us; skew 500 ppm
Sep 26 04:10:35 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:35.584039 16185 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:35.584733 16185 webserver.cc:460] Webserver started at http://127.0.213.254:37245/ using document root <none> and password file <none>
I20260926 04:10:35.584995 16185 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:35.585043 16185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:35.585141 16185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:35.586195 16185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "754dfaa649724a9f9bb3bf7da06c64bd"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "7d7bbc5d42f1ddf053e2bf68fcc0a0f8"
server_key_iv: "7a5f99e9f0a1facb8e95cc4a3a5f0d7f"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.586513 16185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "754dfaa649724a9f9bb3bf7da06c64bd"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "7d7bbc5d42f1ddf053e2bf68fcc0a0f8"
server_key_iv: "7a5f99e9f0a1facb8e95cc4a3a5f0d7f"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.587848 16185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:10:35.588692 16200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:35.588860 16185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:35.588950 16185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "754dfaa649724a9f9bb3bf7da06c64bd"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "7d7bbc5d42f1ddf053e2bf68fcc0a0f8"
server_key_iv: "7a5f99e9f0a1facb8e95cc4a3a5f0d7f"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.589027 16185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:35.599459 16185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:35.601917 16185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:35.602037 16185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:35.605810 16185 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45025
I20260926 04:10:35.605895 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45025 every 8 connection(s)
I20260926 04:10:35.606240 16185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:35.607192 16253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:35.609649 16253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap starting.
I20260926 04:10:35.609694 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16185
I20260926 04:10:35.609810 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:10:35.610024 855 external_mini_cluster.cc:1428] Setting key 5751967768dbf7da79c89542d6ea8ad2
I20260926 04:10:35.610416 16253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:35.611016 16253 log.cc:824] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:35.611816 16253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: No bootstrap required, opened a new log
I20260926 04:10:35.614347 16253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:35.614538 16253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:35.614590 16253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754dfaa649724a9f9bb3bf7da06c64bd, State: Initialized, Role: FOLLOWER
I20260926 04:10:35.614755 16253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:35.614856 16253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:35.614886 16253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:35.614923 16253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:35.615752 16253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:35.615880 16253 leader_election.cc:302] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 754dfaa649724a9f9bb3bf7da06c64bd; no voters:
I20260926 04:10:35.616111 16253 leader_election.cc:288] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:10:35.616148 16258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:35.616273 16258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 LEADER]: Becoming Leader. State: Replica: 754dfaa649724a9f9bb3bf7da06c64bd, State: Running, Role: LEADER
I20260926 04:10:35.616431 16253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:35.616438 16258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:35.616897 16259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } } }
I20260926 04:10:35.616988 16259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:35.617259 16260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 754dfaa649724a9f9bb3bf7da06c64bd. Latest consensus state: current_term: 1 leader_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } } }
I20260926 04:10:35.617396 16260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:35.617491 16265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 04:10:35 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:35.618216 16265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:35.620409 16265 catalog_manager.cc:1347] Generated new cluster ID: 689deb58fec84215b75baebac7596f1d
I20260926 04:10:35.620468 16265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:35.622910 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:35.610669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.0.213.254:45025)
0926 04:10:35.610850 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:35.610861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:35.610896 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:10:35.611427 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:35.611435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:35.611522 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:35.611674 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:35.612407 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:35.613880 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:35.614715 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:35.615346 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:35.615463 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:35.618486 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:35.618490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:35.618490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:35.618504 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:35.620598 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:35.621077 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:35.621081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:35.621081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:35.621192 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:35.621424 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:35.621425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:35.621425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:35.621761 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:35.621929 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:35.622506 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 04:10:35.622672 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 04:10:35.639861 16265 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:10:35.640694 16265 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:35.646665 16265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Generated new TSK 0
I20260926 04:10:35.646909 16265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:10:35.698673 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45025
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:35.793874 16279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:35.794193 16279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:35.794246 16279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:35.796005 16279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:35.796139 16279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:35.796197 16279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:35.798127 16279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:35.798460 16279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:35.798731 16279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:35.804428 16287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:35.804682 16279 server_base.cc:1034] running on GCE node
W20260926 04:10:35.804440 16284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:35.804453 16285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:35.805354 16279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:35.805616 16279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:35.806794 16279 hybrid_clock.cc:648] HybridClock initialized: now 1790395835806734 us; error 82 us; skew 500 ppm
Sep 26 04:10:35 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:35.809885 16279 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:35.810343 16279 webserver.cc:460] Webserver started at http://127.0.213.193:37593/ using document root <none> and password file <none>
I20260926 04:10:35.810565 16279 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:35.810609 16279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:35.810710 16279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:35.811861 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "14565ad960074cf78ddcb39ebaf8188a"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "499df43bf49e648a4dcda57f83b53135"
server_key_iv: "dd8d7a47a743551e780cf2852554d253"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.812292 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "14565ad960074cf78ddcb39ebaf8188a"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "499df43bf49e648a4dcda57f83b53135"
server_key_iv: "dd8d7a47a743551e780cf2852554d253"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.813714 16279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:35.814462 16294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:35.814649 16279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:10:35.814772 16279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "14565ad960074cf78ddcb39ebaf8188a"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "499df43bf49e648a4dcda57f83b53135"
server_key_iv: "dd8d7a47a743551e780cf2852554d253"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.814860 16279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:35.831727 16279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:35.834470 16279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:35.834659 16279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:35.834932 16279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:35.835283 16279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:35.835347 16279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:35.835398 16279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:35.835427 16279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:35.842375 16279 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:41935
I20260926 04:10:35.842551 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:41935 every 8 connection(s)
I20260926 04:10:35.842859 16279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:10:35.845824 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16279
I20260926 04:10:35.845938 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:10:35.846146 855 external_mini_cluster.cc:1428] Setting key 63b7de11deb44ea067e78f55a99f1b1f
Sep 26 04:10:35 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:35.854365 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:35.843701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:36289 (local address 127.0.213.254:45025)
0926 04:10:35.843787 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:35.843790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:35.844004 (+ 214us) server_negotiation.cc:407] Connection header received
0926 04:10:35.844558 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:35.844560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:35.844571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:35.844633 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:35.845678 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:35.846469 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:35.847620 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:35.848013 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:35.848121 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:35.850768 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:35.850770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:35.850770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:35.850778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:35.852655 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:35.853187 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:35.853189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:35.853190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:35.853263 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:35.853573 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:35.853574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:35.853575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:35.853717 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:35.853806 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:35.854153 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 04:10:35.854257 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 04:10:35.854612 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:35.843654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:36289)
0926 04:10:35.843866 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:35.843906 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:35.844428 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:35.844691 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:35.844695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:35.845326 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:35.845556 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:35.845560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:35.846579 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:35.846581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:35.847523 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:35.847527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:35.848190 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:35.848192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:35.848403 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:35.849001 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:35.849013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:35.850644 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:35.852781 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:35.852791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:35.852794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:35.853075 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:35.853348 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:35.853350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:35.853350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:35.853485 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:35.853797 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:35.853805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:35.853996 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 04:10:35.854131 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 04:10:35.854320 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260926 04:10:35.855273 16408 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45025
I20260926 04:10:35.855381 16408 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:35.855602 16408 heartbeater.cc:511] Master 127.0.213.254:45025 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:10:35.856370 16217 ts_manager.cc:194] Registered new tserver with Master: 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935)
I20260926 04:10:35.857193 16217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:36289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:10:35.875489 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45025
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:35.958590 16413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:35.958799 16413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:35.958823 16413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:35.959988 16413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:35.960029 16413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:35.960110 16413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:35.961576 16413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:35.961827 16413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:35.962087 16413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:35.967622 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:35.967674 16413 server_base.cc:1034] running on GCE node
W20260926 04:10:35.967615 16418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:35.967615 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:35.968196 16413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:35.968466 16413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:35.969621 16413 hybrid_clock.cc:648] HybridClock initialized: now 1790395835969607 us; error 27 us; skew 500 ppm
Sep 26 04:10:35 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:35.972506 16413 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:35.973018 16413 webserver.cc:460] Webserver started at http://127.0.213.194:41411/ using document root <none> and password file <none>
I20260926 04:10:35.973268 16413 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:35.973312 16413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:35.973472 16413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:35.974545 16413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "10417eecf95448d59733d00090a8acef"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "344c14a4407d9c8d19126826796cf966"
server_key_iv: "63a2247359abb95b1b32b0b2270260d5"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.974913 16413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "10417eecf95448d59733d00090a8acef"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "344c14a4407d9c8d19126826796cf966"
server_key_iv: "63a2247359abb95b1b32b0b2270260d5"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.976348 16413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:10:35.977106 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:35.977285 16413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:35.977388 16413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "10417eecf95448d59733d00090a8acef"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "344c14a4407d9c8d19126826796cf966"
server_key_iv: "63a2247359abb95b1b32b0b2270260d5"
server_key_version: "encryptionkey@0"
I20260926 04:10:35.977471 16413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:36.012418 16413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:36.014881 16413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:36.015057 16413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:36.015308 16413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:36.015623 16413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:36.015679 16413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:36.015720 16413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:36.015749 16413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:36.022307 16413 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38417
I20260926 04:10:36.022356 16541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38417 every 8 connection(s)
I20260926 04:10:36.022686 16413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:10:36 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:36.031103 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16413
I20260926 04:10:36.031229 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:10:36.031481 855 external_mini_cluster.cc:1428] Setting key 1e663e8e6a57b6a73338420c5346d34c
I20260926 04:10:36.032752 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.023536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:44647 (local address 127.0.213.254:45025)
0926 04:10:36.023612 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.023614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.023811 (+ 197us) server_negotiation.cc:407] Connection header received
0926 04:10:36.024323 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.024324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.024335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.024390 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:36.025358 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.025874 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.026871 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.027059 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.027118 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.029668 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:36.029670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:36.029670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:36.029676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:36.031091 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:36.031636 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:36.031637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:36.031638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:36.031704 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:36.031985 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:36.031986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:36.031987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:36.032152 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:36.032244 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.032569 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.032655 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:10:36.032971 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.023495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:44647)
0926 04:10:36.023681 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.023718 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.024218 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.024446 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.024449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.025029 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:36.025223 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.025227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.025971 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.025972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.026782 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.026784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.027158 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.027159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.027274 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.027879 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:36.027887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:36.029564 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:36.031191 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:36.031200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:36.031202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:36.031526 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:36.031817 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:36.031817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:36.031818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:36.031920 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:36.032248 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:36.032255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:36.032427 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.032548 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.032711 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260926 04:10:36.033676 16542 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45025
I20260926 04:10:36.033802 16542 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:36.034003 16542 heartbeater.cc:511] Master 127.0.213.254:45025 requested a full tablet report, sending...
I20260926 04:10:36.034457 16217 ts_manager.cc:194] Registered new tserver with Master: 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
I20260926 04:10:36.034919 16217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:44647
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:10:36.060289 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45025
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:36.145170 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:36.145401 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:36.145427 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:36.146618 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:36.146711 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:36.146790 16547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:36.148373 16547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:36.148684 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:36.149004 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:10:36.155229 16547 server_base.cc:1034] running on GCE node
W20260926 04:10:36.155236 16553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:36.155275 16552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:36.155486 16555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:36.155737 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:36.156111 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:36.157307 16547 hybrid_clock.cc:648] HybridClock initialized: now 1790395836157299 us; error 36 us; skew 500 ppm
Sep 26 04:10:36 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:36.160025 16547 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:36.160617 16547 webserver.cc:460] Webserver started at http://127.0.213.195:44559/ using document root <none> and password file <none>
I20260926 04:10:36.160851 16547 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:36.160930 16547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:36.161093 16547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:10:36.162240 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "19f4ac1bb3074155a686cb6d96f59093"
format_stamp: "Formatted at 2026-09-26 04:10:36 on dist-test-slave-xf24"
server_key: "eeb6ea48e064ffe0c9b52fadbd1b471d"
server_key_iv: "7612751da070d57bc9c68b6b28aac11e"
server_key_version: "encryptionkey@0"
I20260926 04:10:36.162629 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "19f4ac1bb3074155a686cb6d96f59093"
format_stamp: "Formatted at 2026-09-26 04:10:36 on dist-test-slave-xf24"
server_key: "eeb6ea48e064ffe0c9b52fadbd1b471d"
server_key_iv: "7612751da070d57bc9c68b6b28aac11e"
server_key_version: "encryptionkey@0"
I20260926 04:10:36.164188 16547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:10:36.165038 16562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:36.165246 16547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:36.165352 16547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "19f4ac1bb3074155a686cb6d96f59093"
format_stamp: "Formatted at 2026-09-26 04:10:36 on dist-test-slave-xf24"
server_key: "eeb6ea48e064ffe0c9b52fadbd1b471d"
server_key_iv: "7612751da070d57bc9c68b6b28aac11e"
server_key_version: "encryptionkey@0"
I20260926 04:10:36.165464 16547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:36.197115 16547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:36.199548 16547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:36.199666 16547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:36.199874 16547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:36.200265 16547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:36.200312 16547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:36.200362 16547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:36.200390 16547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:36.206487 16547 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:42055
I20260926 04:10:36.206540 16675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:42055 every 8 connection(s)
I20260926 04:10:36.206874 16547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:10:36 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:36.216365 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16547
I20260926 04:10:36.216466 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:10:36.216468 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.207639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:53739 (local address 127.0.213.254:45025)
0926 04:10:36.207686 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.207688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.207952 (+ 264us) server_negotiation.cc:407] Connection header received
0926 04:10:36.208399 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.208400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.208413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.208482 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:36.209389 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.209886 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.210966 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.211149 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.211201 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.213485 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:36.213486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:36.213487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:36.213493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:36.214773 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:36.215269 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:36.215270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:36.215270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:36.215329 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:36.215615 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:36.215620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:36.215620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:36.215744 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:36.215852 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.216241 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.216340 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:10:36.216615 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.207601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:53739)
0926 04:10:36.207752 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.207805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.208308 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.208536 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.208540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.209104 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:36.209303 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.209308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.210004 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.210005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.210872 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.210874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.211224 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.211225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.211346 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.211925 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:36.211933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:36.213391 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:36.214853 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:36.214861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:36.214863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:36.215101 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:36.215415 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:36.215416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:36.215416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:36.215524 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:36.215819 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:36.215824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:36.216032 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.216212 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.216351 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260926 04:10:36.216694 855 external_mini_cluster.cc:1428] Setting key c49cc062ca4ed5cae39f058797316d37
I20260926 04:10:36.217139 16676 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45025
I20260926 04:10:36.217255 16676 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:36.217451 16676 heartbeater.cc:511] Master 127.0.213.254:45025 requested a full tablet report, sending...
I20260926 04:10:36.217847 16216 ts_manager.cc:194] Registered new tserver with Master: 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055)
I20260926 04:10:36.218318 16216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:53739
I20260926 04:10:36.228910 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:10:36.236677 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.230391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.213.254:45025)
0926 04:10:36.230493 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.230494 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.230504 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:36.230568 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.230568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.230577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.230631 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:36.231216 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.231729 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.232551 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.232736 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.232813 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.234065 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:36.234066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:36.234067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:36.234074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:36.235422 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:36.235749 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:36.235750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:36.235750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:36.235802 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:36.236017 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:36.236018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:36.236018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:36.236234 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:36.236319 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.236483 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.236551 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 04:10:36.238503 16216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:10:36.240032 16216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:10:36.246893 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.242570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.0.213.195:42055)
0926 04:10:36.242744 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.242750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.242762 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:10:36.242836 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.242838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.242855 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.242973 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:36.243158 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.244325 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.245664 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.246411 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246494 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.246559 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.246746 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.246794 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260926 04:10:36.246897 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.242409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41935 (local address 127.0.0.1:42488)
0926 04:10:36.242862 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.242871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.242922 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.243217 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.243218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.243220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:36.243266 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.243267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.244400 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.244401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.245696 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.245698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.246599 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.246681 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.246686 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.246738 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.246809 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":50,"thread_start_us":139,"threads_started":1}
I20260926 04:10:36.246958 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.242680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.213.194:38417)
0926 04:10:36.243111 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.243116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.243447 (+ 331us) server_negotiation.cc:407] Connection header received
0926 04:10:36.243492 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.243494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.243509 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.243581 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:36.243721 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.244770 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246029 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.246605 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246683 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.246742 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.246830 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.246867 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":86,"threads_started":1}
I20260926 04:10:36.247290 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.242843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.213.193:41935)
0926 04:10:36.243043 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.243047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.243054 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:10:36.243086 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.243088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.243098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.243221 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:36.243353 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.244299 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.245796 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.246463 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246521 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.246566 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.247168 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.247210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260926 04:10:36.246908 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.242409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38417 (local address 127.0.0.1:51778)
0926 04:10:36.243314 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.243324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.243374 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.243604 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.243605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.243607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:36.243645 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.243646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.244885 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.244886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.245929 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.245931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.246700 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.246758 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.246761 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.246818 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.246847 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":13,"thread_start_us":124,"threads_started":1}
I20260926 04:10:36.247166 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.242371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42055 (local address 127.0.0.1:34168)
0926 04:10:36.242625 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.242670 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.242739 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.242982 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.242985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.242988 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:36.243064 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.243066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.244457 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.244458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.245577 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.245580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.246517 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.246518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.246610 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.246628 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.247043 (+ 415us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.247094 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":21,"thread_start_us":59,"threads_started":1}
I20260926 04:10:36.248258 16610 tablet_service.cc:1467] Processing CreateTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec (DEFAULT_TABLE table=test-table [id=d5135ebd0a3d40a18245c8ed56160ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:36.248245 16342 tablet_service.cc:1467] Processing CreateTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec (DEFAULT_TABLE table=test-table [id=d5135ebd0a3d40a18245c8ed56160ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:36.248237 16476 tablet_service.cc:1467] Processing CreateTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec (DEFAULT_TABLE table=test-table [id=d5135ebd0a3d40a18245c8ed56160ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:36.248603 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509fb98fd81b41e9a7c5bf727e09fdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:36.248608 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509fb98fd81b41e9a7c5bf727e09fdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:36.248608 16342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509fb98fd81b41e9a7c5bf727e09fdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:36.251267 16693 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Bootstrap starting.
I20260926 04:10:36.251822 16694 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap starting.
I20260926 04:10:36.252189 16693 tablet_bootstrap.cc:654] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:36.252524 16694 tablet_bootstrap.cc:654] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:36.252818 16693 log.cc:824] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:36.253118 16694 log.cc:824] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:36.253430 16695 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Bootstrap starting.
I20260926 04:10:36.253638 16693 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: No bootstrap required, opened a new log
I20260926 04:10:36.253720 16693 ts_tablet_manager.cc:1397] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 04:10:36.254032 16695 tablet_bootstrap.cc:654] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:36.254496 16695 log.cc:824] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:36.255041 16694 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: No bootstrap required, opened a new log
I20260926 04:10:36.255095 16694 ts_tablet_manager.cc:1397] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 04:10:36.255470 16693 raft_consensus.cc:348] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.255633 16693 raft_consensus.cc:374] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:36.255678 16693 raft_consensus.cc:729] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f4ac1bb3074155a686cb6d96f59093, State: Initialized, Role: FOLLOWER
I20260926 04:10:36.255793 16693 consensus_queue.cc:260] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.256141 16693 ts_tablet_manager.cc:1428] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 04:10:36.256181 16676 heartbeater.cc:503] Master 127.0.213.254:45025 was elected leader, sending a full tablet report...
I20260926 04:10:36.256423 16695 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: No bootstrap required, opened a new log
I20260926 04:10:36.256485 16695 ts_tablet_manager.cc:1397] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 04:10:36.257227 16694 raft_consensus.cc:348] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.257397 16694 raft_consensus.cc:374] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:36.257429 16694 raft_consensus.cc:729] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14565ad960074cf78ddcb39ebaf8188a, State: Initialized, Role: FOLLOWER
I20260926 04:10:36.257560 16694 consensus_queue.cc:260] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.257884 16408 heartbeater.cc:503] Master 127.0.213.254:45025 was elected leader, sending a full tablet report...
I20260926 04:10:36.257866 16694 ts_tablet_manager.cc:1428] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:10:36.259167 16695 raft_consensus.cc:348] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.259301 16695 raft_consensus.cc:374] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:36.259323 16695 raft_consensus.cc:729] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10417eecf95448d59733d00090a8acef, State: Initialized, Role: FOLLOWER
I20260926 04:10:36.259404 16695 consensus_queue.cc:260] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.259598 16695 ts_tablet_manager.cc:1428] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:10:36.259665 16542 heartbeater.cc:503] Master 127.0.213.254:45025 was elected leader, sending a full tablet report...
W20260926 04:10:36.273334 16543 tablet.cc:2369] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:10:36.343715 16409 tablet.cc:2369] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:10:36.457468 16677 tablet.cc:2369] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:10:36.500658 16699 raft_consensus.cc:482] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:36.500794 16699 raft_consensus.cc:504] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.501202 16699 leader_election.cc:288] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10417eecf95448d59733d00090a8acef (127.0.213.194:38417), 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935)
I20260926 04:10:36.505395 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.501428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38417 (local address 127.0.213.195:36929)
0926 04:10:36.501477 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.501490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.501547 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.501716 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.501717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.501719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:36.501807 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.501809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.502839 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.502840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.504247 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.504251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.505072 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.505073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.505160 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.505162 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.505223 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.505281 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":9}
I20260926 04:10:36.505425 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.501533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:36929 (local address 127.0.213.194:38417)
0926 04:10:36.501586 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.501587 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.501598 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:10:36.501632 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.501633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.501645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.501729 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:36.501915 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.502714 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.504388 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.504964 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.505054 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.505063 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.505297 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.505351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 04:10:36.505411 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.501485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41935 (local address 127.0.213.195:59757)
0926 04:10:36.501660 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.501669 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.501720 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.501879 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:36.501880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:36.501882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:36.501950 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.501951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.502839 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.502840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.504247 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:36.504251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.505072 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.505073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:36.505155 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.505159 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:10:36.505223 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:10:36.505281 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":92,"thread_start_us":96,"threads_started":1}
I20260926 04:10:36.505470 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.501607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:59757 (local address 127.0.213.193:41935)
0926 04:10:36.501656 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.501657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.501747 (+ 90us) server_negotiation.cc:407] Connection header received
0926 04:10:36.501781 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.501782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.501792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.501846 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:36.502028 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.502720 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.504388 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.504964 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.505054 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.505063 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.505297 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.505354 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:10:36.505801 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509fb98fd81b41e9a7c5bf727e09fdec" candidate_uuid: "19f4ac1bb3074155a686cb6d96f59093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10417eecf95448d59733d00090a8acef" is_pre_election: true
I20260926 04:10:36.505846 16362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509fb98fd81b41e9a7c5bf727e09fdec" candidate_uuid: "19f4ac1bb3074155a686cb6d96f59093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14565ad960074cf78ddcb39ebaf8188a" is_pre_election: true
I20260926 04:10:36.506006 16362 raft_consensus.cc:2454] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f4ac1bb3074155a686cb6d96f59093 in term 0.
I20260926 04:10:36.506011 16496 raft_consensus.cc:2454] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f4ac1bb3074155a686cb6d96f59093 in term 0.
I20260926 04:10:36.506276 16566 leader_election.cc:302] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14565ad960074cf78ddcb39ebaf8188a, 19f4ac1bb3074155a686cb6d96f59093; no voters:
I20260926 04:10:36.506451 16699 raft_consensus.cc:2781] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:10:36.506512 16699 raft_consensus.cc:482] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:10:36.506542 16699 raft_consensus.cc:3037] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:36.507217 16699 raft_consensus.cc:504] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.507369 16699 leader_election.cc:288] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [CANDIDATE]: Term 1 election: Requested vote from peers 10417eecf95448d59733d00090a8acef (127.0.213.194:38417), 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935)
I20260926 04:10:36.507579 16496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509fb98fd81b41e9a7c5bf727e09fdec" candidate_uuid: "19f4ac1bb3074155a686cb6d96f59093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10417eecf95448d59733d00090a8acef"
I20260926 04:10:36.507566 16362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509fb98fd81b41e9a7c5bf727e09fdec" candidate_uuid: "19f4ac1bb3074155a686cb6d96f59093" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14565ad960074cf78ddcb39ebaf8188a"
I20260926 04:10:36.507648 16496 raft_consensus.cc:3037] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:36.507680 16362 raft_consensus.cc:3037] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:36.508494 16496 raft_consensus.cc:2454] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f4ac1bb3074155a686cb6d96f59093 in term 1.
I20260926 04:10:36.508602 16362 raft_consensus.cc:2454] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f4ac1bb3074155a686cb6d96f59093 in term 1.
I20260926 04:10:36.508698 16564 leader_election.cc:302] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093; no voters:
I20260926 04:10:36.508810 16699 raft_consensus.cc:2781] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:36.509021 16699 raft_consensus.cc:686] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 1 LEADER]: Becoming Leader. State: Replica: 19f4ac1bb3074155a686cb6d96f59093, State: Running, Role: LEADER
I20260926 04:10:36.509112 16699 consensus_queue.cc:237] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:36.510157 16217 catalog_manager.cc:5495] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195). New cstate: current_term: 1 leader_uuid: "19f4ac1bb3074155a686cb6d96f59093" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260926 04:10:36.535523 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:36.532921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.0.213.195:42055)
0926 04:10:36.532994 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:36.532996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:36.533004 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:10:36.533049 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:36.533050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:36.533068 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:36.533167 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:36.533335 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.533859 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.534695 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:36.534903 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:36.534986 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:36.535177 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:36.535281 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:10:36.535384 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:36.535403 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 04:10:36.535424 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 04:10:36.537551 16496 raft_consensus.cc:1261] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Refusing update from remote peer 19f4ac1bb3074155a686cb6d96f59093: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:36.537600 16362 raft_consensus.cc:1261] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Refusing update from remote peer 19f4ac1bb3074155a686cb6d96f59093: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:36.537976 16703 consensus_queue.cc:1035] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:36.538149 16699 consensus_queue.cc:1035] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:36.543294 16216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:36.543442 16216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:36.544412 16216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Sending DeleteTablet for 3 replicas of tablet 509fb98fd81b41e9a7c5bf727e09fdec
I20260926 04:10:36.544885 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16279
I20260926 04:10:36.544870 16610 tablet_service.cc:1514] Processing DeleteTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:36 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:34168
I20260926 04:10:36.544977 16342 tablet_service.cc:1514] Processing DeleteTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:36 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:42488
I20260926 04:10:36.546365 16476 tablet_service.cc:1514] Processing DeleteTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:36 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:51778
I20260926 04:10:36.546432 16716 tablet_replica.cc:330] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: stopping tablet replica
I20260926 04:10:36.546568 16716 raft_consensus.cc:2229] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:10:36.546687 16716 raft_consensus.cc:2258] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:36.547230 16716 ts_tablet_manager.cc:1905] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:36.548461 16717 tablet_replica.cc:330] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: stopping tablet replica
I20260926 04:10:36.548635 16717 raft_consensus.cc:2229] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:36.548804 16717 raft_consensus.cc:2258] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:36.549305 16717 ts_tablet_manager.cc:1905] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:36.549659 16716 ts_tablet_manager.cc:1918] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:36.549746 16716 log.cc:1196] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/509fb98fd81b41e9a7c5bf727e09fdec
I20260926 04:10:36.550093 16716 ts_tablet_manager.cc:1939] T 509fb98fd81b41e9a7c5bf727e09fdec P 19f4ac1bb3074155a686cb6d96f59093: Deleting consensus metadata
I20260926 04:10:36.550601 16201 catalog_manager.cc:4841] TS 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): tablet 509fb98fd81b41e9a7c5bf727e09fdec (table test-table [id=d5135ebd0a3d40a18245c8ed56160ca3]) successfully deleted
I20260926 04:10:36.551568 16717 ts_tablet_manager.cc:1918] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:36.551704 16717 log.cc:1196] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/509fb98fd81b41e9a7c5bf727e09fdec
W20260926 04:10:36.551851 16204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:41935 (error 108)
I20260926 04:10:36.552114 16717 ts_tablet_manager.cc:1939] T 509fb98fd81b41e9a7c5bf727e09fdec P 10417eecf95448d59733d00090a8acef: Deleting consensus metadata
I20260926 04:10:36.552606 16202 catalog_manager.cc:4841] TS 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): tablet 509fb98fd81b41e9a7c5bf727e09fdec (table test-table [id=d5135ebd0a3d40a18245c8ed56160ca3]) successfully deleted
W20260926 04:10:36.552706 16687 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:10:36.552313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41935 (local address 127.0.0.1:42490)
0926 04:10:36.552393 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:36.552407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:36.552453 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:36.552607 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:41935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 04:10:36.552824 16204 catalog_manager.cc:4584] TS 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 509fb98fd81b41e9a7c5bf727e09fdec: Network error: Client connection negotiation failed: client connection to 127.0.213.193:41935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:10:36.552908 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16413
I20260926 04:10:36.559787 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16547
I20260926 04:10:36.564247 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16185
I20260926 04:10:56.570152 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45025
--webserver_interface=127.0.213.254
--webserver_port=37245
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:10:56.651903 16719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:56.652168 16719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:56.652192 16719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:56.653438 16719 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 04:10:56.653488 16719 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 04:10:56.653502 16719 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 04:10:56.653514 16719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:10:56.653527 16719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:56.653540 16719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:56.653599 16719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:10:56.653617 16719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:10:56.655161 16719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:56.655434 16719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:56.655704 16719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:56.664935 16726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:56.665017 16725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:56.664935 16728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:56.665040 16719 server_base.cc:1034] running on GCE node
I20260926 04:10:56.665433 16719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:56.665737 16719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:56.666911 16719 hybrid_clock.cc:648] HybridClock initialized: now 1790395856666885 us; error 49 us; skew 500 ppm
Sep 26 04:10:56 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:56.676409 16719 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:10:56.677024 16719 webserver.cc:460] Webserver started at http://127.0.213.254:37245/ using document root <none> and password file <none>
I20260926 04:10:56.677299 16719 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:56.677387 16719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:56.678957 16719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:10:56.679733 16735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:56.679958 16719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:56.680099 16719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "754dfaa649724a9f9bb3bf7da06c64bd"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "7d7bbc5d42f1ddf053e2bf68fcc0a0f8"
server_key_iv: "7a5f99e9f0a1facb8e95cc4a3a5f0d7f"
server_key_version: "encryptionkey@0"
I20260926 04:10:56.680542 16719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:56.702126 16719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:56.704694 16719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:56.704885 16719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:56.709609 16719 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45025
I20260926 04:10:56.709646 16787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45025 every 8 connection(s)
I20260926 04:10:56.710043 16719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:10:56.711627 16788 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:10:56.712332 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap starting.
I20260926 04:10:56.715688 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16719
I20260926 04:10:56.716105 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:41935
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:45025
--webserver_port=37593
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:56.721746 16788 log.cc:824] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:56.723690 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:56.724042 16788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap complete.
I20260926 04:10:56.726382 16788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:56.726610 16788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754dfaa649724a9f9bb3bf7da06c64bd, State: Initialized, Role: FOLLOWER
I20260926 04:10:56.726778 16788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:56.726848 16788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:10:56.726907 16788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:10:56.726958 16788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:10:56.727964 16788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:56.728099 16788 leader_election.cc:302] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 754dfaa649724a9f9bb3bf7da06c64bd; no voters:
I20260926 04:10:56.728317 16788 leader_election.cc:288] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:10:56.728351 16792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:10:56.728469 16792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 LEADER]: Becoming Leader. State: Replica: 754dfaa649724a9f9bb3bf7da06c64bd, State: Running, Role: LEADER
I20260926 04:10:56.728606 16788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:10:56.728621 16792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:10:56.729130 16793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } } }
I20260926 04:10:56.729218 16793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:56.729475 16794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 754dfaa649724a9f9bb3bf7da06c64bd. Latest consensus state: current_term: 2 leader_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } } }
I20260926 04:10:56.729578 16794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: This master's current role is: LEADER
I20260926 04:10:56.729628 16799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:10:56.730545 16799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:10:56.730695 16799 catalog_manager.cc:1259] Loaded cluster ID: 689deb58fec84215b75baebac7596f1d
I20260926 04:10:56.730880 16799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:10:56.732236 16799 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:10:56.732432 16799 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Loaded TSK: 0
I20260926 04:10:56.751654 16799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Generated new TSK 1
I20260926 04:10:56.751799 16799 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:10:56.812736 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:56.812965 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:56.812994 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:56.814373 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:56.814432 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:56.814484 16790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:10:56.816186 16790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:41935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=37593
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:56.816435 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:56.816774 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:56.823484 16815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:56.823508 16818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:56.823510 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:56.823746 16790 server_base.cc:1034] running on GCE node
I20260926 04:10:56.824204 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:56.824460 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:56.825632 16790 hybrid_clock.cc:648] HybridClock initialized: now 1790395856825618 us; error 33 us; skew 500 ppm
Sep 26 04:10:56 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:56.829092 16790 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:10:56.829667 16790 webserver.cc:460] Webserver started at http://127.0.213.193:37593/ using document root <none> and password file <none>
I20260926 04:10:56.829929 16790 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:56.830029 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:56.831854 16790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:10:56.832777 16825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:56.833029 16790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:10:56.833127 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "14565ad960074cf78ddcb39ebaf8188a"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "499df43bf49e648a4dcda57f83b53135"
server_key_iv: "dd8d7a47a743551e780cf2852554d253"
server_key_version: "encryptionkey@0"
I20260926 04:10:56.833602 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:56.861472 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:56.864261 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:56.864414 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:56.864655 16790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:56.865113 16832 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:10:56.866698 16790 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:10:56.866755 16790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:10:56.866789 16790 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:10:56.867612 16790 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:10:56.867651 16790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:10:56.867769 16832 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap starting.
I20260926 04:10:56.874891 16790 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:41935
I20260926 04:10:56.874984 16939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:41935 every 8 connection(s)
I20260926 04:10:56.875322 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:10:56.878659 16832 log.cc:824] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:56.881245 16832 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:10:56.881577 16832 tablet_bootstrap.cc:492] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap complete.
I20260926 04:10:56.882234 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16790
I20260926 04:10:56.882527 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:38417
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:45025
--webserver_port=41411
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:10:56.882699 16832 ts_tablet_manager.cc:1397] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260926 04:10:56.884298 16832 raft_consensus.cc:348] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:56.884595 16832 raft_consensus.cc:729] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14565ad960074cf78ddcb39ebaf8188a, State: Initialized, Role: FOLLOWER
I20260926 04:10:56.884758 16832 consensus_queue.cc:260] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:56.885072 16832 ts_tablet_manager.cc:1428] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Sep 26 04:10:56 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:56.889791 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:56.876186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:45935)
0926 04:10:56.876404 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:56.876450 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:56.876955 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:56.879379 (+ 2424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:56.879382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:56.879629 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:56.879837 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:56.879843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.881496 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.881497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:56.882402 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:56.882405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.883196 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.883198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:56.883344 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:56.883957 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:56.883968 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:56.885732 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:56.888157 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:56.888164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:56.888166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:56.888390 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:56.888696 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:56.888697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:56.888697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:56.888802 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:56.889312 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:56.889316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:56.889439 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 04:10:56.889523 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 04:10:56.889649 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":133,"threads_started":1}
I20260926 04:10:56.890022 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:56.876268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:45935 (local address 127.0.213.254:45025)
0926 04:10:56.878922 (+ 2654us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:56.878932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:56.878982 (+ 50us) server_negotiation.cc:407] Connection header received
0926 04:10:56.879101 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:56.879106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:56.879174 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:56.879342 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:56.879977 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.881391 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.882503 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.883069 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.883204 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:56.885867 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:56.885871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:56.885872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:56.885885 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:56.888045 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:56.888487 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:56.888491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:56.888492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:56.888618 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:56.888873 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:56.888874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:56.888874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:56.889229 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:56.889331 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:56.889610 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 04:10:56.889783 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2570,"thread_start_us":108,"threads_started":1}
I20260926 04:10:56.890727 16940 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45025
I20260926 04:10:56.890817 16940 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:56.891027 16940 heartbeater.cc:511] Master 127.0.213.254:45025 requested a full tablet report, sending...
I20260926 04:10:56.891765 16752 ts_manager.cc:194] Registered new tserver with Master: 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935)
I20260926 04:10:56.892872 16752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:45935
I20260926 04:10:56.893957 16940 heartbeater.cc:503] Master 127.0.213.254:45025 was elected leader, sending a full tablet report...
Sep 26 04:10:56 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.193@KRBTEST.COM
I20260926 04:10:56.903931 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:56.892477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41935 (local address 127.0.0.1:58230)
0926 04:10:56.893208 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:56.893243 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:56.893305 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:56.893611 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:56.893618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:56.894028 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:56.894106 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:56.894109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.895597 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.895599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:56.896427 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:56.896431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.897260 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.897261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:56.897350 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:56.898110 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:56.898117 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:56.899903 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:56.902402 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:56.902415 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:56.902417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:56.902707 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:56.903006 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:56.903008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:56.903008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:56.903086 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:56.903499 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:56.903511 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:56.903578 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 04:10:56.903673 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 04:10:56.903843 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":102,"threads_started":1}
I20260926 04:10:56.903954 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:56.892543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.213.193:41935)
0926 04:10:56.893448 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:56.893453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:56.893463 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:56.893496 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:56.893499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:56.893512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:56.893583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:56.894223 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.895470 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.896532 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:56.897083 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:56.897173 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:56.900038 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:56.900075 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:56.900075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:56.900086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:56.902212 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:56.902813 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:56.902817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:56.902817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:56.902897 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:56.903162 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:56.903163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:56.903163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:56.903382 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:56.903469 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:56.903735 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 04:10:56.903843 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":82,"threads_started":1}
I20260926 04:10:56.904711 16874 tablet_service.cc:1514] Processing DeleteTablet for tablet 509fb98fd81b41e9a7c5bf727e09fdec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:36 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:58230
I20260926 04:10:56.905368 16949 tablet_replica.cc:330] stopping tablet replica
I20260926 04:10:56.905505 16949 raft_consensus.cc:2229] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:56.905591 16949 raft_consensus.cc:2258] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:56.905942 16949 ts_tablet_manager.cc:1905] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:56.908380 16949 ts_tablet_manager.cc:1918] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:56.908469 16949 log.cc:1196] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal/wals/509fb98fd81b41e9a7c5bf727e09fdec
I20260926 04:10:56.908720 16949 ts_tablet_manager.cc:1939] T 509fb98fd81b41e9a7c5bf727e09fdec P 14565ad960074cf78ddcb39ebaf8188a: Deleting consensus metadata
I20260926 04:10:56.909420 16739 catalog_manager.cc:4841] TS 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935): tablet 509fb98fd81b41e9a7c5bf727e09fdec (table test-table [id=d5135ebd0a3d40a18245c8ed56160ca3]) successfully deleted
W20260926 04:10:56.976662 16945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:56.976913 16945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:56.976949 16945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:56.978091 16945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:56.978161 16945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:56.978209 16945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:10:56.979781 16945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:38417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=41411
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:56.980052 16945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:56.980298 16945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:56.985494 16957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:56.985517 16954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:56.985587 16945 server_base.cc:1034] running on GCE node
W20260926 04:10:56.985517 16955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:56.985903 16945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:56.986171 16945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:56.987331 16945 hybrid_clock.cc:648] HybridClock initialized: now 1790395856987321 us; error 24 us; skew 500 ppm
Sep 26 04:10:56 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:56.990049 16945 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:10:56.990538 16945 webserver.cc:460] Webserver started at http://127.0.213.194:41411/ using document root <none> and password file <none>
I20260926 04:10:56.990767 16945 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:56.990823 16945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:56.992275 16945 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:56.992956 16964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:56.993204 16945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:56.993310 16945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "10417eecf95448d59733d00090a8acef"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "344c14a4407d9c8d19126826796cf966"
server_key_iv: "63a2247359abb95b1b32b0b2270260d5"
server_key_version: "encryptionkey@0"
I20260926 04:10:56.993743 16945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:57.024735 16945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:57.027185 16945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:57.027338 16945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:57.027621 16945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:57.027951 16945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:57.028013 16945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:57.028049 16945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:57.028103 16945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:57.034708 16945 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38417
I20260926 04:10:57.034746 17077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38417 every 8 connection(s)
I20260926 04:10:57.035141 16945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:10:57.039963 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 16945
I20260926 04:10:57.040280 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:42055
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:45025
--webserver_port=44559
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:10:57 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:57.045351 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.035967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:54739 (local address 127.0.213.254:45025)
0926 04:10:57.036041 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.036043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.036310 (+ 267us) server_negotiation.cc:407] Connection header received
0926 04:10:57.036755 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.036756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.036770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.036846 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:57.037873 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.038485 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.039453 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.039665 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.039731 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.042284 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:57.042286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:57.042286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:57.042294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:57.043848 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.044347 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.044349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.044350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.044411 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.044676 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.044677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.044678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.044792 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:57.044871 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.045161 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.045260 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:10:57.045549 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.035954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:54739)
0926 04:10:57.036134 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.036177 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:57.036668 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:57.036906 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:57.036909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:57.037547 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:57.037747 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.037752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.038574 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.038574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.039374 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.039376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.039820 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.039822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.039936 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.040593 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:57.040602 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:57.042165 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:57.043939 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:57.043946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:57.043949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:57.044256 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:57.044480 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:57.044481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:57.044482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:57.044598 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:57.044868 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:57.044875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:57.045022 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 04:10:57.045128 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 04:10:57.045291 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260926 04:10:57.046221 17078 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45025
I20260926 04:10:57.046357 17078 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:57.046630 17078 heartbeater.cc:511] Master 127.0.213.254:45025 requested a full tablet report, sending...
I20260926 04:10:57.047076 16752 ts_manager.cc:194] Registered new tserver with Master: 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
I20260926 04:10:57.047482 16752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:54739
W20260926 04:10:57.127506 17081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:10:57.127733 17081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:10:57.127760 17081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:10:57.129305 17081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:10:57.129362 17081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:10:57.129395 17081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:10:57.131493 17081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:42055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=44559
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:10:57.131807 17081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:10:57.132114 17081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:10:57.137979 17087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:10:57.137979 17086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:57.138046 17081 server_base.cc:1034] running on GCE node
W20260926 04:10:57.138031 17089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:10:57.138490 17081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:10:57.138808 17081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:10:57.139999 17081 hybrid_clock.cc:648] HybridClock initialized: now 1790395857139985 us; error 31 us; skew 500 ppm
Sep 26 04:10:57 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:10:57.142774 17081 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:10:57.143211 17081 webserver.cc:460] Webserver started at http://127.0.213.195:44559/ using document root <none> and password file <none>
I20260926 04:10:57.143496 17081 fs_manager.cc:359] Metadata directory not provided
I20260926 04:10:57.143556 17081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:10:57.145165 17081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:57.146013 17096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:57.146353 17081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:57.146507 17081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "19f4ac1bb3074155a686cb6d96f59093"
format_stamp: "Formatted at 2026-09-26 04:10:36 on dist-test-slave-xf24"
server_key: "eeb6ea48e064ffe0c9b52fadbd1b471d"
server_key_iv: "7612751da070d57bc9c68b6b28aac11e"
server_key_version: "encryptionkey@0"
I20260926 04:10:57.146934 17081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:10:57.159780 17081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:10:57.162335 17081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:10:57.162494 17081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:10:57.162779 17081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:10:57.163127 17081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:10:57.163187 17081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:57.163228 17081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:10:57.163257 17081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:10:57.169854 17081 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:42055
I20260926 04:10:57.169900 17209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:42055 every 8 connection(s)
I20260926 04:10:57.170274 17081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:10:57.175951 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17081
Sep 26 04:10:57 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:10:57.180825 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.171141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:48063 (local address 127.0.213.254:45025)
0926 04:10:57.171232 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.171234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.171402 (+ 168us) server_negotiation.cc:407] Connection header received
0926 04:10:57.171884 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.171885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.171895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.171961 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:57.172954 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.173461 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.174518 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.174715 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.174783 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.177479 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:57.177480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:57.177480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:57.177487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:57.179135 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.179771 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.179773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.179773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.179841 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.180105 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.180107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.180107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.180230 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:57.180312 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.180607 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.180701 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:10:57.180984 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.171105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:48063)
0926 04:10:57.171258 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.171303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:57.171790 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:57.172014 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:57.172018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:57.172626 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:10:57.172837 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.172841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.173562 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.173563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.174406 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.174414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.174822 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.174823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.174977 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.175512 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:10:57.175521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:10:57.177349 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:10:57.179230 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:57.179237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:57.179239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:57.179686 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:57.179903 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:10:57.179904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:10:57.179904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:10:57.180017 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:10:57.180303 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:10:57.180309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:10:57.180455 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 04:10:57.180568 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 04:10:57.180739 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260926 04:10:57.181627 17210 heartbeater.cc:347] Connected to a master server at 127.0.213.254:45025
I20260926 04:10:57.181756 17210 heartbeater.cc:464] Registering TS with master...
I20260926 04:10:57.181968 17210 heartbeater.cc:511] Master 127.0.213.254:45025 requested a full tablet report, sending...
I20260926 04:10:57.182464 16752 ts_manager.cc:194] Registered new tserver with Master: 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055)
I20260926 04:10:57.182894 16752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:48063
I20260926 04:10:57.183624 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.176439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.213.254:45025)
0926 04:10:57.176575 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.176577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.176651 (+ 74us) server_negotiation.cc:407] Connection header received
0926 04:10:57.176705 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.176706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.176721 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.176787 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:57.177432 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.177930 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.178879 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.179231 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.179296 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.180742 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:57.180743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:57.180744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:57.180751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:57.182457 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.182824 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.182825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.182825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.182885 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.183087 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.183088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.183088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.183192 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:57.183373 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.183437 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.183530 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260926 04:10:57.184456 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:10:57.187445 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.184798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.0.213.254:45025)
0926 04:10:57.184850 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.184853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.184980 (+ 127us) server_negotiation.cc:407] Connection header received
0926 04:10:57.185081 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.185086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.185115 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.185191 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:57.185290 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.185785 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.186558 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.186749 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.186823 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.187001 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:57.187047 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 04:10:57.187343 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38672: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:10:57.187512 17214 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38672: token expired
W20260926 04:10:57.187711 16682 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.213.254:45025)
I20260926 04:10:57.197161 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.188121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.0.213.254:45025)
0926 04:10:57.188200 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.188202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.188212 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:57.188244 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.188245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.188256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.188326 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:10:57.189069 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.189963 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.191147 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.191366 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.191422 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.193489 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:10:57.193491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:10:57.193491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:10:57.193499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:10:57.195512 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.195956 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.195957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.195957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.196022 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:10:57.196473 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:10:57.196474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:10:57.196474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:10:57.196612 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:10:57.196698 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.196949 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.197050 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 04:10:57.198659 16752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:10:57.200335 16752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:10:57.202836 16874 tablet_service.cc:1467] Processing CreateTablet for tablet 9461215edb254dd2afc56e6f3447ffc8 (DEFAULT_TABLE table=test-table [id=51ead8d768db41c5bc36c5ecf9c6938a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:57.203048 16874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9461215edb254dd2afc56e6f3447ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:57.204674 16832 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap starting.
I20260926 04:10:57.205044 16832 tablet_bootstrap.cc:654] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:57.205724 16832 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: No bootstrap required, opened a new log
I20260926 04:10:57.205786 16832 ts_tablet_manager.cc:1397] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:57.206144 16832 raft_consensus.cc:348] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.206228 16832 raft_consensus.cc:374] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:57.206267 16832 raft_consensus.cc:729] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14565ad960074cf78ddcb39ebaf8188a, State: Initialized, Role: FOLLOWER
I20260926 04:10:57.206312 16832 consensus_queue.cc:260] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.206404 16832 ts_tablet_manager.cc:1428] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:10:57.207177 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.202525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42055 (local address 127.0.0.1:40294)
0926 04:10:57.202822 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.202838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:57.202890 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:57.203176 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:57.203176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:57.203179 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:57.203267 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.203270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.204610 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.204611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.205886 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.205890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.206898 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.206899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.206960 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.206973 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 04:10:57.207050 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 04:10:57.207097 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":5,"thread_start_us":61,"threads_started":1}
I20260926 04:10:57.207188 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.202614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.213.195:42055)
0926 04:10:57.202794 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.202798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.202899 (+ 101us) server_negotiation.cc:407] Connection header received
0926 04:10:57.202946 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.202948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.202963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.203059 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:57.203359 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.204506 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.205996 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.206721 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.206794 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.206860 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.207067 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.207109 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260926 04:10:57.208146 17144 tablet_service.cc:1467] Processing CreateTablet for tablet 9461215edb254dd2afc56e6f3447ffc8 (DEFAULT_TABLE table=test-table [id=51ead8d768db41c5bc36c5ecf9c6938a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:57.208510 17144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9461215edb254dd2afc56e6f3447ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:57.208854 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.202663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.213.194:38417)
0926 04:10:57.204767 (+ 2104us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.204772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.204782 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:57.204828 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.204831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.204850 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.204969 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:57.205095 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.206356 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.207830 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.208506 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.208564 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.208638 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.208753 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.208787 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2062,"thread_start_us":77,"threads_started":1}
I20260926 04:10:57.208853 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.202423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38417 (local address 127.0.0.1:49694)
0926 04:10:57.202479 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.202497 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:57.202538 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:57.204973 (+ 2435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:57.204974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:57.204976 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:57.205014 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.205015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.206673 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.206674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.207741 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.207744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.208591 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.208592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.208667 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.208671 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:10:57.208735 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:10:57.208782 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:10:57.209630 17012 tablet_service.cc:1467] Processing CreateTablet for tablet 9461215edb254dd2afc56e6f3447ffc8 (DEFAULT_TABLE table=test-table [id=51ead8d768db41c5bc36c5ecf9c6938a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:10:57.209975 17012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9461215edb254dd2afc56e6f3447ffc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:10:57.212246 17220 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Bootstrap starting.
I20260926 04:10:57.212823 17220 tablet_bootstrap.cc:654] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:57.213313 17220 log.cc:824] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:57.213409 17221 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Bootstrap starting.
I20260926 04:10:57.213995 17221 tablet_bootstrap.cc:654] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Neither blocks nor log segments found. Creating new log.
I20260926 04:10:57.214175 17220 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: No bootstrap required, opened a new log
I20260926 04:10:57.214257 17220 ts_tablet_manager.cc:1397] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:10:57.214629 17221 log.cc:824] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Log is configured to *not* fsync() on all Append() calls
I20260926 04:10:57.215399 17221 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: No bootstrap required, opened a new log
I20260926 04:10:57.215539 17221 ts_tablet_manager.cc:1397] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:10:57.216204 17220 raft_consensus.cc:348] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.216351 17220 raft_consensus.cc:374] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:57.216380 17220 raft_consensus.cc:729] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f4ac1bb3074155a686cb6d96f59093, State: Initialized, Role: FOLLOWER
I20260926 04:10:57.216518 17220 consensus_queue.cc:260] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.216815 17220 ts_tablet_manager.cc:1428] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:10:57.216848 17210 heartbeater.cc:503] Master 127.0.213.254:45025 was elected leader, sending a full tablet report...
I20260926 04:10:57.217370 17221 raft_consensus.cc:348] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.217583 17221 raft_consensus.cc:374] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:10:57.217623 17221 raft_consensus.cc:729] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10417eecf95448d59733d00090a8acef, State: Initialized, Role: FOLLOWER
I20260926 04:10:57.217737 17221 consensus_queue.cc:260] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.218026 17221 ts_tablet_manager.cc:1428] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:10:57.218104 17078 heartbeater.cc:503] Master 127.0.213.254:45025 was elected leader, sending a full tablet report...
I20260926 04:10:57.243533 17225 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:10:57.243677 17225 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.244213 17225 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935)
I20260926 04:10:57.247488 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.244561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:46581 (local address 127.0.213.193:41935)
0926 04:10:57.244609 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.244611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.244716 (+ 105us) server_negotiation.cc:407] Connection header received
0926 04:10:57.244788 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.244789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.244818 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.244892 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:57.245054 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.245545 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.246682 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.247118 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.247222 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.247308 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.247356 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.247409 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:10:57.247509 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.244519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:39645 (local address 127.0.213.195:42055)
0926 04:10:57.244581 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.244583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.244593 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:10:57.244624 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.244625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.244642 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.244725 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:10:57.244901 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.245394 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.246700 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.247114 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.247222 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.247233 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.247388 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.247439 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:10:57.247514 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.244429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42055 (local address 127.0.213.194:39645)
0926 04:10:57.244470 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.244484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:57.244538 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:57.244734 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:57.244736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:57.244737 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:57.244817 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.244819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.245480 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.245481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.246503 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.246506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.247208 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.247209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.247297 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.247300 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:10:57.247378 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 04:10:57.247420 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":1}
I20260926 04:10:57.247526 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.244444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41935 (local address 127.0.213.194:46581)
0926 04:10:57.244637 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.244645 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:10:57.244722 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:10:57.244898 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:10:57.244899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:10:57.244900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:10:57.244971 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.244973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.245632 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.245633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.246504 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:10:57.246507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.247222 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.247223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:10:57.247289 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.247294 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:10:57.247350 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:10:57.247393 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260926 04:10:57.247911 17164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461215edb254dd2afc56e6f3447ffc8" candidate_uuid: "10417eecf95448d59733d00090a8acef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f4ac1bb3074155a686cb6d96f59093" is_pre_election: true
I20260926 04:10:57.248133 17164 raft_consensus.cc:2454] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10417eecf95448d59733d00090a8acef in term 0.
I20260926 04:10:57.248064 16894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461215edb254dd2afc56e6f3447ffc8" candidate_uuid: "10417eecf95448d59733d00090a8acef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14565ad960074cf78ddcb39ebaf8188a" is_pre_election: true
I20260926 04:10:57.248255 16894 raft_consensus.cc:2454] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10417eecf95448d59733d00090a8acef in term 0.
I20260926 04:10:57.248427 16965 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093; no voters:
I20260926 04:10:57.248570 17225 raft_consensus.cc:2781] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:10:57.248632 17225 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:10:57.248651 17225 raft_consensus.cc:3037] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:57.249580 17225 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.249730 17225 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [CANDIDATE]: Term 1 election: Requested vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935)
I20260926 04:10:57.249903 17164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461215edb254dd2afc56e6f3447ffc8" candidate_uuid: "10417eecf95448d59733d00090a8acef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f4ac1bb3074155a686cb6d96f59093"
I20260926 04:10:57.249971 17164 raft_consensus.cc:3037] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:57.249915 16894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461215edb254dd2afc56e6f3447ffc8" candidate_uuid: "10417eecf95448d59733d00090a8acef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14565ad960074cf78ddcb39ebaf8188a"
I20260926 04:10:57.250029 16894 raft_consensus.cc:3037] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:10:57.250684 17164 raft_consensus.cc:2454] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10417eecf95448d59733d00090a8acef in term 1.
I20260926 04:10:57.250885 16965 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093; no voters:
I20260926 04:10:57.251019 17225 raft_consensus.cc:2781] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:10:57.251101 16894 raft_consensus.cc:2454] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10417eecf95448d59733d00090a8acef in term 1.
I20260926 04:10:57.251243 17225 raft_consensus.cc:686] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 1 LEADER]: Becoming Leader. State: Replica: 10417eecf95448d59733d00090a8acef, State: Running, Role: LEADER
I20260926 04:10:57.251350 17225 consensus_queue.cc:237] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:10:57.252406 16751 catalog_manager.cc:5495] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef reported cstate change: term changed from 0 to 1, leader changed from <none> to 10417eecf95448d59733d00090a8acef (127.0.213.194). New cstate: current_term: 1 leader_uuid: "10417eecf95448d59733d00090a8acef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260926 04:10:57.266709 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.264285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.0.213.194:38417)
0926 04:10:57.264349 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:10:57.264351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:10:57.264376 (+ 25us) server_negotiation.cc:407] Connection header received
0926 04:10:57.264421 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:10:57.264422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:10:57.264439 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:10:57.264492 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:10:57.264594 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.265087 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.265852 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:10:57.266070 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:10:57.266140 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:10:57.266309 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:10:57.266420 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:10:57.266499 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:10:57.266535 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 04:10:57.266557 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:10:57.269032 17164 raft_consensus.cc:1261] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Refusing update from remote peer 10417eecf95448d59733d00090a8acef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:57.269070 16894 raft_consensus.cc:1261] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Refusing update from remote peer 10417eecf95448d59733d00090a8acef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:10:57.269445 17227 consensus_queue.cc:1035] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:57.269613 17225 consensus_queue.cc:1035] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [LEADER]: Connected to new peer: Peer: permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:10:57.272342 17232 mvcc.cc:205] Tried to move back new op lower bound from 7333461431370866688 to 7333461431302447104. Current Snapshot: MvccSnapshot[applied={T|T < 7333461431370866688}]
I20260926 04:10:57.272557 17231 mvcc.cc:205] Tried to move back new op lower bound from 7333461431370866688 to 7333461431302447104. Current Snapshot: MvccSnapshot[applied={T|T < 7333461431370866688}]
I20260926 04:10:57.274600 16752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:57.274729 16752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:10:57.275549 16752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Sending DeleteTablet for 3 replicas of tablet 9461215edb254dd2afc56e6f3447ffc8
I20260926 04:10:57.275915 17012 tablet_service.cc:1514] Processing DeleteTablet for tablet 9461215edb254dd2afc56e6f3447ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:57 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:49694
I20260926 04:10:57.276096 17144 tablet_service.cc:1514] Processing DeleteTablet for tablet 9461215edb254dd2afc56e6f3447ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:57 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40294
I20260926 04:10:57.276202 17239 tablet_replica.cc:330] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: stopping tablet replica
I20260926 04:10:57.276232 16874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9461215edb254dd2afc56e6f3447ffc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:10:57 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:58230
I20260926 04:10:57.276321 16949 tablet_replica.cc:330] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: stopping tablet replica
I20260926 04:10:57.276320 17239 raft_consensus.cc:2229] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:10:57.276336 17240 tablet_replica.cc:330] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: stopping tablet replica
I20260926 04:10:57.276364 16949 raft_consensus.cc:2229] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:57.276363 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16790
I20260926 04:10:57.276419 16949 raft_consensus.cc:2258] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:57.276433 17239 raft_consensus.cc:2258] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:57.277230 17240 raft_consensus.cc:2229] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:10:57.277374 17240 raft_consensus.cc:2258] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:10:57.277642 17239 ts_tablet_manager.cc:1905] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:57.277921 17240 ts_tablet_manager.cc:1905] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:10:57.279829 17239 ts_tablet_manager.cc:1918] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:57.279858 17240 ts_tablet_manager.cc:1918] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:10:57.279924 17239 log.cc:1196] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/9461215edb254dd2afc56e6f3447ffc8
I20260926 04:10:57.279968 17240 log.cc:1196] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/9461215edb254dd2afc56e6f3447ffc8
I20260926 04:10:57.280262 17239 ts_tablet_manager.cc:1939] T 9461215edb254dd2afc56e6f3447ffc8 P 10417eecf95448d59733d00090a8acef: Deleting consensus metadata
I20260926 04:10:57.280297 17240 ts_tablet_manager.cc:1939] T 9461215edb254dd2afc56e6f3447ffc8 P 19f4ac1bb3074155a686cb6d96f59093: Deleting consensus metadata
I20260926 04:10:57.280817 16736 catalog_manager.cc:4841] TS 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): tablet 9461215edb254dd2afc56e6f3447ffc8 (table test-table [id=51ead8d768db41c5bc36c5ecf9c6938a]) successfully deleted
I20260926 04:10:57.280901 16737 catalog_manager.cc:4841] TS 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): tablet 9461215edb254dd2afc56e6f3447ffc8 (table test-table [id=51ead8d768db41c5bc36c5ecf9c6938a]) successfully deleted
W20260926 04:10:57.283039 16739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:41935 (error 108)
I20260926 04:10:57.283262 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16945
I20260926 04:10:57.283962 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:10:57.283386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:41935 (local address 127.0.0.1:58234)
0926 04:10:57.283878 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 04:10:57.283914 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:41935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472}
W20260926 04:10:57.284134 16739 catalog_manager.cc:4584] TS 14565ad960074cf78ddcb39ebaf8188a (127.0.213.193:41935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9461215edb254dd2afc56e6f3447ffc8: Network error: Client connection negotiation failed: client connection to 127.0.213.193:41935: connect: Connection refused (error 111)
I20260926 04:10:57.288765 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 17081
I20260926 04:10:57.293560 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 16719
I20260926 04:11:17.299381 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45025
--webserver_interface=127.0.213.254
--webserver_port=37245
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:17.382730 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:17.382987 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:17.383011 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:17.384275 17242 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 04:11:17.384332 17242 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 04:11:17.384346 17242 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 04:11:17.384358 17242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:17.384372 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:17.384385 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:17.384404 17242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:17.384418 17242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:17.385931 17242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:17.386142 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:17.386323 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:17.391795 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.391803 17242 server_base.cc:1034] running on GCE node
W20260926 04:11:17.391820 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:17.391808 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.392283 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:17.392622 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:17.393776 17242 hybrid_clock.cc:648] HybridClock initialized: now 1790395877393757 us; error 43 us; skew 500 ppm
Sep 26 04:11:17 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:17.402807 17242 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:11:17.403378 17242 webserver.cc:460] Webserver started at http://127.0.213.254:37245/ using document root <none> and password file <none>
I20260926 04:11:17.403662 17242 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:17.403749 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:17.405347 17242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.406260 17257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.406533 17242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.406651 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "754dfaa649724a9f9bb3bf7da06c64bd"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "7d7bbc5d42f1ddf053e2bf68fcc0a0f8"
server_key_iv: "7a5f99e9f0a1facb8e95cc4a3a5f0d7f"
server_key_version: "encryptionkey@0"
I20260926 04:11:17.407054 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:17.414736 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:17.417454 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:17.417641 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:17.422379 17242 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45025
I20260926 04:11:17.422415 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45025 every 8 connection(s)
I20260926 04:11:17.422827 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:17.424536 17310 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:11:17.424870 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17242
I20260926 04:11:17.425173 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:41935
--local_ip_for_outbound_sockets=127.0.213.193
--tserver_master_addrs=127.0.213.254:45025
--webserver_port=37593
--webserver_interface=127.0.213.193
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:11:17.425426 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap starting.
I20260926 04:11:17.434818 17310 log.cc:824] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:17.437427 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:11:17.437796 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Bootstrap complete.
I20260926 04:11:17.440189 17310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:11:17.440431 17310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754dfaa649724a9f9bb3bf7da06c64bd, State: Initialized, Role: FOLLOWER
I20260926 04:11:17.440600 17310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:11:17.440660 17310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:17.440723 17310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:17.440765 17310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 2 FOLLOWER]: Advancing to term 3
I20260926 04:11:17.441769 17310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:11:17.441859 17310 leader_election.cc:302] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 754dfaa649724a9f9bb3bf7da06c64bd; no voters:
I20260926 04:11:17.442083 17310 leader_election.cc:288] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 04:11:17.442137 17314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 3 FOLLOWER]: Leader election won for term 3
I20260926 04:11:17.442260 17314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [term 3 LEADER]: Becoming Leader. State: Replica: 754dfaa649724a9f9bb3bf7da06c64bd, State: Running, Role: LEADER
I20260926 04:11:17.442353 17314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } }
I20260926 04:11:17.442384 17310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:17.442761 17316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 754dfaa649724a9f9bb3bf7da06c64bd. Latest consensus state: current_term: 3 leader_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } } }
I20260926 04:11:17.442852 17316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:17.442970 17315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "754dfaa649724a9f9bb3bf7da06c64bd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45025 } } }
I20260926 04:11:17.443109 17315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:17.443128 17318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:17.443972 17318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:17.444120 17318 catalog_manager.cc:1259] Loaded cluster ID: 689deb58fec84215b75baebac7596f1d
I20260926 04:11:17.444159 17318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:17.445333 17318 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:17.445515 17318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Loaded TSK: 0
I20260926 04:11:17.445565 17318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 754dfaa649724a9f9bb3bf7da06c64bd: Loaded TSK: 1
F20260926 04:11:17.445804 17318 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7fe5c549ad google::LogMessage::Fail() at ??:0
@ 0x7f7fe5c5788a google::LogMessage::SendToLog() at ??:0
@ 0x7f7fe5c544de google::LogMessage::Flush() at ??:0
@ 0x7f7fe5c55a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7fe83c1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7fe83bed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7fedbb21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7fedbb248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7fedbb2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7fedb78c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7fedb82ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7fe7b220d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7fe7b17954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7fe897b6db start_thread at ??:0
@ 0x7f7fe475261f clone at ??:0
W20260926 04:11:17.536831 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:17.537043 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:17.537068 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:17.538298 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:17.538340 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:17.538367 17312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:17.539822 17312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:41935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=37593
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:17.540059 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:17.540323 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:17.545570 17354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:17.545687 17352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.545630 17312 server_base.cc:1034] running on GCE node
W20260926 04:11:17.545571 17351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.546069 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:17.546342 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:17.547501 17312 hybrid_clock.cc:648] HybridClock initialized: now 1790395877547478 us; error 41 us; skew 500 ppm
Sep 26 04:11:17 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:17.550201 17312 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:11:17.550684 17312 webserver.cc:460] Webserver started at http://127.0.213.193:37593/ using document root <none> and password file <none>
I20260926 04:11:17.550909 17312 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:17.550984 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:17.552377 17312 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.553117 17361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.553332 17312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.553416 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "14565ad960074cf78ddcb39ebaf8188a"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "499df43bf49e648a4dcda57f83b53135"
server_key_iv: "dd8d7a47a743551e780cf2852554d253"
server_key_version: "encryptionkey@0"
I20260926 04:11:17.553787 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:17.561506 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:17.563881 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:17.564040 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:17.564378 17312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:17.564965 17368 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:11:17.566473 17312 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:11:17.566527 17312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:11:17.566565 17312 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:11:17.567268 17312 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:11:17.567312 17312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:11:17.567360 17368 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap starting.
I20260926 04:11:17.575102 17312 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:41935
I20260926 04:11:17.575654 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:11:17.576216 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:41935 every 8 connection(s)
I20260926 04:11:17.578682 17368 log.cc:824] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:17.580960 17368 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:11:17.581022 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17312
I20260926 04:11:17.581278 17368 tablet_bootstrap.cc:492] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Bootstrap complete.
I20260926 04:11:17.581300 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:38417
--local_ip_for_outbound_sockets=127.0.213.194
--tserver_master_addrs=127.0.213.254:45025
--webserver_port=41411
--webserver_interface=127.0.213.194
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:11:17.582463 17368 ts_tablet_manager.cc:1397] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260926 04:11:17.583501 17368 raft_consensus.cc:348] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:17.583746 17368 raft_consensus.cc:729] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14565ad960074cf78ddcb39ebaf8188a, State: Initialized, Role: FOLLOWER
I20260926 04:11:17.583920 17368 consensus_queue.cc:260] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:17.584260 17368 ts_tablet_manager.cc:1428] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260926 04:11:17.650007 17478 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:11:17.576381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:32845)
0926 04:11:17.576591 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.576658 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:17.577409 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:17.649827 (+ 72418us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
W20260926 04:11:17.650317 17476 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45025 (0 consecutive failures): Network error: Failed to ping master at 127.0.213.254:45025: Client connection negotiation failed: client connection to 127.0.213.254:45025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:11:17.650779 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.650604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:44207)
0926 04:11:17.650688 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.650736 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 04:11:17.651206 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.651117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:35039)
0926 04:11:17.651162 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.651186 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 04:11:17.651336 17476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:11:17.667840 17480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:17.668097 17480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:17.668141 17480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:17.669306 17480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:17.669345 17480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:17.669371 17480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:17.670764 17480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:38417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=41411
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:17.670944 17480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:17.671123 17480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:17.676192 17489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:17.676216 17487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:17.676254 17486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.676347 17480 server_base.cc:1034] running on GCE node
I20260926 04:11:17.676760 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:17.676978 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:17.678128 17480 hybrid_clock.cc:648] HybridClock initialized: now 1790395877678108 us; error 36 us; skew 500 ppm
Sep 26 04:11:17 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:17.680739 17480 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:11:17.681254 17480 webserver.cc:460] Webserver started at http://127.0.213.194:41411/ using document root <none> and password file <none>
I20260926 04:11:17.681492 17480 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:17.681562 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:17.682958 17480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.683589 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.683782 17480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.683904 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "10417eecf95448d59733d00090a8acef"
format_stamp: "Formatted at 2026-09-26 04:10:35 on dist-test-slave-xf24"
server_key: "344c14a4407d9c8d19126826796cf966"
server_key_iv: "63a2247359abb95b1b32b0b2270260d5"
server_key_version: "encryptionkey@0"
I20260926 04:11:17.684342 17480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:17.696935 17480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:17.698935 17480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:17.699115 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:17.699368 17480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:17.699685 17480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:17.699740 17480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.699775 17480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:17.699821 17480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.705878 17480 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:38417
I20260926 04:11:17.705920 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:38417 every 8 connection(s)
I20260926 04:11:17.706272 17480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:17.707056 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17480
I20260926 04:11:17.707302 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.706916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:42979)
0926 04:11:17.707076 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.707166 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 04:11:17.707438 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:42055
--local_ip_for_outbound_sockets=127.0.213.195
--tserver_master_addrs=127.0.213.254:45025
--webserver_port=44559
--webserver_interface=127.0.213.195
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:17.707665 17610 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45025 (0 consecutive failures): Network error: Failed to ping master at 127.0.213.254:45025: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
I20260926 04:11:17.707937 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.707836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:56761)
0926 04:11:17.707892 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.707920 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:11:17.708225 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.708134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:58213)
0926 04:11:17.708178 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.708203 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 04:11:17.708312 17610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:11:17.791471 17613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:17.791672 17613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:17.791697 17613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:17.792975 17613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:17.793020 17613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:17.793046 17613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:17.794500 17613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:42055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=44559
--tserver_master_addrs=127.0.213.254:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:17.794754 17613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:17.795002 17613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:17.800071 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:17.799993 17621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.800089 17613 server_base.cc:1034] running on GCE node
W20260926 04:11:17.800235 17618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:17.800581 17613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:17.800827 17613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:17.801986 17613 hybrid_clock.cc:648] HybridClock initialized: now 1790395877801971 us; error 32 us; skew 500 ppm
Sep 26 04:11:17 dist-test-slave-xf24 krb5kdc[16172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:17.804593 17613 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:11:17.805012 17613 webserver.cc:460] Webserver started at http://127.0.213.195:44559/ using document root <none> and password file <none>
I20260926 04:11:17.805253 17613 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:17.805307 17613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:17.806690 17613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.807447 17628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.807614 17613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:17.807710 17613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "19f4ac1bb3074155a686cb6d96f59093"
format_stamp: "Formatted at 2026-09-26 04:10:36 on dist-test-slave-xf24"
server_key: "eeb6ea48e064ffe0c9b52fadbd1b471d"
server_key_iv: "7612751da070d57bc9c68b6b28aac11e"
server_key_version: "encryptionkey@0"
I20260926 04:11:17.808099 17613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:17.824625 17613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:17.826983 17613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:17.827113 17613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:17.827303 17613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:17.827629 17613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:17.827693 17613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.827737 17613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:17.827764 17613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:17.834012 17613 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:42055
I20260926 04:11:17.834052 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:42055 every 8 connection(s)
I20260926 04:11:17.834409 17613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:11:17.835614 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.835143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:51619)
0926 04:11:17.835382 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.835456 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
W20260926 04:11:17.836045 17742 heartbeater.cc:644] Failed to heartbeat to 127.0.213.254:45025 (0 consecutive failures): Network error: Failed to ping master at 127.0.213.254:45025: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
I20260926 04:11:17.836387 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.836278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:48067)
0926 04:11:17.836334 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.836365 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 04:11:17.836774 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:17.836663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:34217)
0926 04:11:17.836721 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:17.836751 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:11:17.836931 17742 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:11:17.842890 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17613
I20260926 04:11:18.652022 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:18.651786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:34805)
0926 04:11:18.651941 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:18.651978 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260926 04:11:18.708873 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:18.708639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:55499)
0926 04:11:18.708783 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:18.708820 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260926 04:11:18.837503 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:18.837258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:59853)
0926 04:11:18.837409 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:18.837448 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260926 04:11:19.424625 17749 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:19.424768 17749 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:19.425140 17749 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
Sep 26 04:11:19 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.195@KRBTEST.COM
Sep 26 04:11:19 dist-test-slave-xf24 krb5kdc[16172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.194@KRBTEST.COM
I20260926 04:11:19.436169 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.425610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:33465 (local address 127.0.213.194:38417)
0926 04:11:19.425797 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:19.425805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:19.425825 (+ 20us) server_negotiation.cc:407] Connection header received
0926 04:11:19.426281 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:19.426284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:19.426327 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:19.426523 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:19.427490 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.428406 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.429495 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.429963 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.430046 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:19.432915 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:19.432917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:19.432918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:19.432949 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:19.434670 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:19.435050 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:19.435053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:19.435053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:19.435140 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:19.435349 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:19.435350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:19.435350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:19.435566 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:19.435659 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:19.435905 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 04:11:19.436029 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260926 04:11:19.436157 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.425416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:38417 (local address 127.0.213.193:33465)
0926 04:11:19.425604 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:19.425619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:19.425679 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:19.426561 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:19.426564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:19.427064 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:19.427344 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:19.427352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.428508 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.428510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:19.429378 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:19.429381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.430053 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.430054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:19.430197 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:19.430802 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:19.430811 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:19.432806 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:19.434779 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:19.434780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:19.434781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:19.434967 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:19.435218 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:19.435219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:19.435220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:19.435282 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:19.435644 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:19.435644 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:19.435728 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 04:11:19.435849 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 04:11:19.436040 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":14,"thread_start_us":90,"threads_started":1}
I20260926 04:11:19.436224 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.425539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:45205 (local address 127.0.213.195:42055)
0926 04:11:19.425702 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:19.425714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:19.425844 (+ 130us) server_negotiation.cc:407] Connection header received
0926 04:11:19.426355 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:19.426358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:19.426414 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:19.426558 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:19.427462 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.428540 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.429478 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.430103 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.430222 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:19.432563 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:19.432568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:19.432568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:19.432581 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:19.434386 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:19.434851 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:19.434855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:19.434855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:19.434952 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:19.435236 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:19.435237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:19.435238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:19.435488 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:19.435586 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:19.435907 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 04:11:19.436064 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
I20260926 04:11:19.436306 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.425445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:42055 (local address 127.0.213.193:45205)
0926 04:11:19.425750 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:19.425761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:19.425853 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:19.426608 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:19.426609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:19.427064 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:19.427344 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:19.427352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.428647 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.428648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:19.429379 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:19.429382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:19.430191 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:19.430192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:19.430234 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:19.430869 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:19.430873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:19.432438 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:19.434500 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:19.434507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:19.434510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:19.434749 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:19.435037 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:19.435038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:19.435038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:19.435154 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:19.435579 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:19.435585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:19.435727 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 04:11:19.435848 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 04:11:19.436164 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":169,"thread_start_us":66,"threads_started":1}
W20260926 04:11:19.437634 17363 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
W20260926 04:11:19.437733 17362 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
I20260926 04:11:19.437783 17362 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14565ad960074cf78ddcb39ebaf8188a; no voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093
I20260926 04:11:19.437902 17749 raft_consensus.cc:2726] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:11:19.652819 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.652624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:56651)
0926 04:11:19.652720 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:19.652765 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 04:11:19.709623 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.709372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:45287)
0926 04:11:19.709520 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:19.709577 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260926 04:11:19.838176 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:19.837965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:37257)
0926 04:11:19.838085 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:19.838131 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260926 04:11:20.653734 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:20.653498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:32923)
0926 04:11:20.653638 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:20.653690 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260926 04:11:20.710716 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:20.710294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:33653)
0926 04:11:20.710457 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:20.710557 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260926 04:11:20.839007 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:20.838701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:49543)
0926 04:11:20.838894 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:20.838950 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260926 04:11:21.619127 17759 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:21.619295 17759 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:21.619524 17759 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
W20260926 04:11:21.620013 17362 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
W20260926 04:11:21.620105 17363 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
I20260926 04:11:21.620177 17363 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14565ad960074cf78ddcb39ebaf8188a; no voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093
I20260926 04:11:21.620241 17759 raft_consensus.cc:2726] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:11:21.654695 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:21.654437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:55743)
0926 04:11:21.654597 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:21.654646 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260926 04:11:21.712045 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:21.711796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:49893)
0926 04:11:21.711938 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:21.711969 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260926 04:11:21.839859 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:21.839641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:60265)
0926 04:11:21.839761 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:21.839796 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 04:11:22.655851 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:22.655522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:39917)
0926 04:11:22.655681 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:22.655794 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260926 04:11:22.712857 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:22.712611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:60089)
0926 04:11:22.712768 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:22.712801 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260926 04:11:22.840728 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:22.840488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:50433)
0926 04:11:22.840614 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:22.840670 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260926 04:11:23.656673 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:23.656449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:59335)
0926 04:11:23.656578 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:23.656625 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 04:11:23.713626 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:23.713396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:45963)
0926 04:11:23.713533 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:23.713579 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 04:11:23.841691 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:23.841406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:52133)
0926 04:11:23.841596 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:23.841644 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260926 04:11:24.464449 17769 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:24.464566 17769 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:24.464746 17769 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
W20260926 04:11:24.465188 17363 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
W20260926 04:11:24.465310 17362 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
I20260926 04:11:24.465396 17362 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14565ad960074cf78ddcb39ebaf8188a; no voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093
I20260926 04:11:24.465485 17769 raft_consensus.cc:2726] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:11:24.657440 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:24.657180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:60715)
0926 04:11:24.657322 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:24.657362 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260926 04:11:24.714432 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:24.714156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:33525)
0926 04:11:24.714332 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:24.714382 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260926 04:11:24.842486 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:24.842240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:60287)
0926 04:11:24.842393 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:24.842438 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260926 04:11:25.658179 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:25.657943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:37871)
0926 04:11:25.658092 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:25.658133 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260926 04:11:25.715237 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:25.715010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:34687)
0926 04:11:25.715146 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:25.715189 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260926 04:11:25.843320 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:25.843039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:46253)
0926 04:11:25.843221 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:25.843266 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260926 04:11:26.615265 17776 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:26.615362 17776 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:26.615595 17776 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
W20260926 04:11:26.615998 17362 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
W20260926 04:11:26.616128 17363 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
I20260926 04:11:26.616195 17363 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14565ad960074cf78ddcb39ebaf8188a; no voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093
I20260926 04:11:26.616295 17776 raft_consensus.cc:2726] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:11:26.659018 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:26.658753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:49235)
0926 04:11:26.658926 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:26.658972 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260926 04:11:26.716032 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:26.715796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:37051)
0926 04:11:26.715942 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:26.715985 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260926 04:11:26.844223 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:26.843909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:59853)
0926 04:11:26.844116 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:26.844172 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260926 04:11:27.659952 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:27.659684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:46621)
0926 04:11:27.659852 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:27.659900 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260926 04:11:27.716832 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:27.716584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:33241)
0926 04:11:27.716737 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:27.716782 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260926 04:11:27.845137 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:27.844894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:45809)
0926 04:11:27.845046 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:27.845080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260926 04:11:28.660877 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:28.660626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:57041)
0926 04:11:28.660764 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:28.660811 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260926 04:11:28.717530 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:28.717325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:59397)
0926 04:11:28.717447 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:28.717479 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260926 04:11:28.845908 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:28.845667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:47691)
0926 04:11:28.845814 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:28.845858 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260926 04:11:29.490590 17786 raft_consensus.cc:482] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:29.490693 17786 raft_consensus.cc:504] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f4ac1bb3074155a686cb6d96f59093" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 42055 } } peers { permanent_uuid: "10417eecf95448d59733d00090a8acef" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 38417 } } peers { permanent_uuid: "14565ad960074cf78ddcb39ebaf8188a" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 41935 } }
I20260926 04:11:29.490862 17786 leader_election.cc:288] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055), 10417eecf95448d59733d00090a8acef (127.0.213.194:38417)
W20260926 04:11:29.491291 17363 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10417eecf95448d59733d00090a8acef (127.0.213.194:38417): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
W20260926 04:11:29.491437 17362 leader_election.cc:341] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f4ac1bb3074155a686cb6d96f59093 (127.0.213.195:42055): Not found: Tablet not found: 9461215edb254dd2afc56e6f3447ffc8
I20260926 04:11:29.491501 17362 leader_election.cc:302] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14565ad960074cf78ddcb39ebaf8188a; no voters: 10417eecf95448d59733d00090a8acef, 19f4ac1bb3074155a686cb6d96f59093
I20260926 04:11:29.491592 17786 raft_consensus.cc:2726] T 9461215edb254dd2afc56e6f3447ffc8 P 14565ad960074cf78ddcb39ebaf8188a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:11:29.661728 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:29.661445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:55361)
0926 04:11:29.661625 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:29.661673 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260926 04:11:29.718525 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:29.718246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:50169)
0926 04:11:29.718432 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:29.718476 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":121,"threads_started":1}
I20260926 04:11:29.846963 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:29.846716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:33157)
0926 04:11:29.846865 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:29.846913 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260926 04:11:30.662679 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:30.662399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:59523)
0926 04:11:30.662586 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:30.662632 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260926 04:11:30.719331 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:30.719105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:37183)
0926 04:11:30.719238 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:30.719282 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260926 04:11:30.847870 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:30.847634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:34095)
0926 04:11:30.847773 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:30.847820 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260926 04:11:31.663503 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:31.663241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:41401)
0926 04:11:31.663419 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:31.663457 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260926 04:11:31.720158 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:31.719834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:57431)
0926 04:11:31.719984 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:31.720046 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260926 04:11:31.848670 17795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:31.848451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:34681)
0926 04:11:31.848582 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:31.848620 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260926 04:11:32.664554 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:32.664282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.193:35567)
0926 04:11:32.664449 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:32.664500 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260926 04:11:32.720943 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:32.720741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.194:33667)
0926 04:11:32.720865 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:32.720897 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260926 04:11:32.847563 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 17312
I20260926 04:11:32.849565 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:32.849321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:45025 (local address 127.0.213.195:44889)
0926 04:11:32.849476 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:32.849521 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.254:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260926 04:11:32.852893 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 17480
I20260926 04:11:32.857693 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 17613
2026-09-26T04:11:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58590 ms)
[----------] 30 tests from SecurityITest (238745 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:32 dist-test-slave-xf24 krb5kdc[17801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:32 dist-test-slave-xf24 krb5kdc[17801](info): set up 2 sockets
Sep 26 04:11:32 dist-test-slave-xf24 krb5kdc[17801](info): commencing operation
krb5kdc: starting...
W20260926 04:11:33.903818 855 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:33 dist-test-slave-xf24 krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:11:34.032611 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36463
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:34.116777 17814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:34.117060 17814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:34.117102 17814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:34.118314 17814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:34.118407 17814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:34.118434 17814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:34.118456 17814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:34.118484 17814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:34.120229 17814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:36463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:34.120546 17814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:34.120805 17814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:34.126724 17819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.126899 17822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.126777 17820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.126760 17814 server_base.cc:1034] running on GCE node
I20260926 04:11:34.127225 17814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:34.127575 17814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:34.128788 17814 hybrid_clock.cc:648] HybridClock initialized: now 1790395894128724 us; error 82 us; skew 500 ppm
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:34.131520 17814 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:11:34.132035 17814 webserver.cc:460] Webserver started at http://127.0.213.254:35817/ using document root <none> and password file <none>
I20260926 04:11:34.132331 17814 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:34.132402 17814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:34.132553 17814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:34.133714 17814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "9a9e4c1446c6480dabd2a0530d96b0da"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "aac8bbd0245ebf66654d5f7614d2dfe0"
server_key_iv: "22c89b49091bdd9e602b02f5b245b50e"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.134130 17814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "9a9e4c1446c6480dabd2a0530d96b0da"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "aac8bbd0245ebf66654d5f7614d2dfe0"
server_key_iv: "22c89b49091bdd9e602b02f5b245b50e"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.135840 17814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:34.136904 17829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.137120 17814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:34.137239 17814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "9a9e4c1446c6480dabd2a0530d96b0da"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "aac8bbd0245ebf66654d5f7614d2dfe0"
server_key_iv: "22c89b49091bdd9e602b02f5b245b50e"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.137333 17814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:34.152664 17814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:34.155287 17814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:34.155481 17814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:34.159917 17814 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:36463
I20260926 04:11:34.159982 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:36463 every 8 connection(s)
I20260926 04:11:34.160391 17814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:34.161425 17882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:34.163784 17882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da: Bootstrap starting.
I20260926 04:11:34.164479 17882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:34.164736 17882 log.cc:824] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:34.165485 17882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da: No bootstrap required, opened a new log
I20260926 04:11:34.167075 17882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36463 } }
I20260926 04:11:34.167238 17882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:34.167263 17882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9e4c1446c6480dabd2a0530d96b0da, State: Initialized, Role: FOLLOWER
I20260926 04:11:34.167409 17882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36463 } }
I20260926 04:11:34.167493 17882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:34.167519 17882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:34.167549 17882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:34.168201 17882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36463 } }
I20260926 04:11:34.168321 17882 leader_election.cc:302] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a9e4c1446c6480dabd2a0530d96b0da; no voters:
I20260926 04:11:34.168336 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17814
I20260926 04:11:34.168437 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:34.168530 17882 leader_election.cc:288] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:34.168684 17886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:34.168725 855 external_mini_cluster.cc:1428] Setting key 80e291fa0e74954c4f67755c3ef8f5ca
I20260926 04:11:34.168814 17882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:34.168814 17886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [term 1 LEADER]: Becoming Leader. State: Replica: 9a9e4c1446c6480dabd2a0530d96b0da, State: Running, Role: LEADER
I20260926 04:11:34.169049 17886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36463 } }
I20260926 04:11:34.169466 17886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a9e4c1446c6480dabd2a0530d96b0da. Latest consensus state: current_term: 1 leader_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36463 } } }
I20260926 04:11:34.169548 17886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:34.169674 17887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a9e4c1446c6480dabd2a0530d96b0da" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36463 } } }
I20260926 04:11:34.169765 17887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:34.169817 17896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:34.170411 17896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:34.172701 17896 catalog_manager.cc:1347] Generated new cluster ID: 44b0447a57984a6d9db89b124dd67375
I20260926 04:11:34.172775 17896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:34.182061 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.169261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.0.213.254:36463)
0926 04:11:34.169867 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.169879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.169927 (+ 48us) server_negotiation.cc:407] Connection header received
0926 04:11:34.170269 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.170280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.170361 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.171324 (+ 963us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:34.171445 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.172839 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.173969 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.174745 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.174882 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.177925 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:34.177931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:34.177932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:34.177949 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:34.180259 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.180702 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.180705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.180705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.180853 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.181065 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.181066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.181066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.181410 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:34.181591 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.181668 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.181823 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":65,"threads_started":1}
I20260926 04:11:34.201117 17896 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:34.201898 17896 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:34.214960 17896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da: Generated new TSK 0
I20260926 04:11:34.215231 17896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:11:34.249711 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36463
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:34.333604 17908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:34.333876 17908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:34.333925 17908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:34.335352 17908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:34.335433 17908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:34.335481 17908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:34.337157 17908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:34.337445 17908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:34.337688 17908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:34.343703 17916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.343650 17914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.343683 17908 server_base.cc:1034] running on GCE node
W20260926 04:11:34.343650 17913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.344357 17908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:34.344673 17908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:34.345862 17908 hybrid_clock.cc:648] HybridClock initialized: now 1790395894345851 us; error 31 us; skew 500 ppm
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:34.348589 17908 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:11:34.349073 17908 webserver.cc:460] Webserver started at http://127.0.213.193:42785/ using document root <none> and password file <none>
I20260926 04:11:34.349316 17908 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:34.349360 17908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:34.349534 17908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:34.350618 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "38fc4b59bae842fbb910451ad283cc5c"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "b894918b4e58bd13d274400b4d20fee8"
server_key_iv: "4362044f20bc0ccb500a0ee447d39e1c"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.350982 17908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "38fc4b59bae842fbb910451ad283cc5c"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "b894918b4e58bd13d274400b4d20fee8"
server_key_iv: "4362044f20bc0ccb500a0ee447d39e1c"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.352387 17908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:11:34.353192 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.353399 17908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:34.353528 17908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "38fc4b59bae842fbb910451ad283cc5c"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "b894918b4e58bd13d274400b4d20fee8"
server_key_iv: "4362044f20bc0ccb500a0ee447d39e1c"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.353603 17908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:34.363893 17908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:34.366070 17908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:34.366222 17908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:34.366516 17908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:34.366833 17908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:34.366886 17908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.366927 17908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:34.366956 17908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.373606 17908 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:43525
I20260926 04:11:34.373667 18036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:43525 every 8 connection(s)
I20260926 04:11:34.374063 17908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:11:34.375447 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 17908
I20260926 04:11:34.375542 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:11:34.375753 855 external_mini_cluster.cc:1428] Setting key 92bebba164729739f85e6a21670ad4c2
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:11:34.385828 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.374945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:52617 (local address 127.0.213.254:36463)
0926 04:11:34.375034 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.375037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.375277 (+ 240us) server_negotiation.cc:407] Connection header received
0926 04:11:34.376444 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.376445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.376459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.376571 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:34.377272 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.377934 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.379100 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.379459 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.379540 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.382204 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:34.382206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:34.382207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:34.382216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:34.383997 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.384544 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.384546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.384546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.384617 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.384949 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.384950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.384950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.385076 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:34.385136 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.385677 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.385761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 04:11:34.385856 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.374916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36463 (local address 127.0.213.193:52617)
0926 04:11:34.375090 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.375136 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.376338 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.376673 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.376678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.376955 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:34.377163 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.377167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.378038 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.378039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.379011 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.379014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.379597 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.379598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.379730 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.380382 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:34.380391 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:34.382094 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:34.384170 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:34.384177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:34.384180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:34.384458 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:34.384740 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:34.384741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:34.384741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:34.384867 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:34.385165 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:34.385171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:34.385304 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.385420 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.385569 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260926 04:11:34.386431 18037 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36463
I20260926 04:11:34.386554 18037 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:34.386806 18037 heartbeater.cc:511] Master 127.0.213.254:36463 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
I20260926 04:11:34.387497 17846 ts_manager.cc:194] Registered new tserver with Master: 38fc4b59bae842fbb910451ad283cc5c (127.0.213.193:43525)
I20260926 04:11:34.388257 17846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:52617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:11:34.403681 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36463
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:34.485977 18042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:34.486186 18042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:34.486212 18042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:34.487385 18042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:34.487427 18042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:34.487454 18042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:34.488960 18042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:34.489204 18042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:34.489449 18042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:34.494706 18047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.494704 18048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.494725 18050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.494943 18042 server_base.cc:1034] running on GCE node
I20260926 04:11:34.495260 18042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:34.495522 18042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:34.496682 18042 hybrid_clock.cc:648] HybridClock initialized: now 1790395894496661 us; error 41 us; skew 500 ppm
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:34.499450 18042 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:11:34.499917 18042 webserver.cc:460] Webserver started at http://127.0.213.194:33071/ using document root <none> and password file <none>
I20260926 04:11:34.500193 18042 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:34.500242 18042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:34.500335 18042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:34.501348 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "865fc0e5977c47b382a362c9d2818252"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "7aaa883e32b557aaae68a46e78baba1e"
server_key_iv: "1410c08c07efc9fa9c92fa4018900c68"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.501644 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "865fc0e5977c47b382a362c9d2818252"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "7aaa883e32b557aaae68a46e78baba1e"
server_key_iv: "1410c08c07efc9fa9c92fa4018900c68"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.502924 18042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:11:34.503722 18057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.503886 18042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:34.503974 18042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "865fc0e5977c47b382a362c9d2818252"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "7aaa883e32b557aaae68a46e78baba1e"
server_key_iv: "1410c08c07efc9fa9c92fa4018900c68"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.504032 18042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:34.519047 18042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:34.521426 18042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:34.521552 18042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:34.521747 18042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:34.522083 18042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:34.522120 18042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.522145 18042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:34.522156 18042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.527915 18042 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:39703
I20260926 04:11:34.528018 18170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:39703 every 8 connection(s)
I20260926 04:11:34.528354 18042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:34.528833 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18042
I20260926 04:11:34.528940 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:11:34.529215 855 external_mini_cluster.cc:1428] Setting key 5080a214189f7d8084428e4452909034
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:11:34.540211 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.529250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:37343 (local address 127.0.213.254:36463)
0926 04:11:34.529311 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.529314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.529806 (+ 492us) server_negotiation.cc:407] Connection header received
0926 04:11:34.531206 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.531207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.531218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.531276 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:34.532109 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.532902 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.534124 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.534473 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.534557 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.537060 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:34.537061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:34.537062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:34.537069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:34.538656 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.539059 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.539060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.539060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.539114 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.539378 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.539379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.539380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.539553 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:34.539615 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.539999 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.540134 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 04:11:34.540365 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.529174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36463 (local address 127.0.213.194:37343)
0926 04:11:34.529461 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.529505 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.531086 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.531360 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.531365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.531767 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:34.531987 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.531992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.533038 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.533040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.534005 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.534008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.534572 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.534573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.534684 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.535293 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:34.535302 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:34.536956 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:34.538729 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:34.538736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:34.538738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:34.538988 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:34.539194 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:34.539195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:34.539196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:34.539310 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:34.539631 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:34.539637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:34.539801 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.539895 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.540122 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260926 04:11:34.540931 18171 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36463
I20260926 04:11:34.541066 18171 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
I20260926 04:11:34.541311 18171 heartbeater.cc:511] Master 127.0.213.254:36463 requested a full tablet report, sending...
I20260926 04:11:34.541817 17846 ts_manager.cc:194] Registered new tserver with Master: 865fc0e5977c47b382a362c9d2818252 (127.0.213.194:39703)
I20260926 04:11:34.542227 17846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:37343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:11:34.559830 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36463
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:34.646741 18176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:34.646999 18176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:34.647029 18176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:34.648476 18176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:34.648531 18176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:34.648564 18176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:34.650305 18176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:34.650576 18176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:34.650859 18176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:34.656639 18184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.656646 18181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.656922 18176 server_base.cc:1034] running on GCE node
W20260926 04:11:34.656670 18182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.657198 18176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:34.657424 18176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:34.658612 18176 hybrid_clock.cc:648] HybridClock initialized: now 1790395894658607 us; error 38 us; skew 500 ppm
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:34.661401 18176 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:11:34.661923 18176 webserver.cc:460] Webserver started at http://127.0.213.195:40451/ using document root <none> and password file <none>
I20260926 04:11:34.662181 18176 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:34.662246 18176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:34.662348 18176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:34.663478 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "336452ef46874f7a9685fb1b39ccc2fa"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "ff0aae913ce78f951b853ae00f80a03e"
server_key_iv: "82c68b6e5fd0764a8678e8ba2c112334"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.663802 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "336452ef46874f7a9685fb1b39ccc2fa"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "ff0aae913ce78f951b853ae00f80a03e"
server_key_iv: "82c68b6e5fd0764a8678e8ba2c112334"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.665293 18176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:34.666088 18191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.666452 18176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:34.666527 18176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "336452ef46874f7a9685fb1b39ccc2fa"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "ff0aae913ce78f951b853ae00f80a03e"
server_key_iv: "82c68b6e5fd0764a8678e8ba2c112334"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.666615 18176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:34.681681 18176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:34.684182 18176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:34.684305 18176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:34.684504 18176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:34.684841 18176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:34.684876 18176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.684929 18176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:34.684952 18176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.691126 18176 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:46143
I20260926 04:11:34.691164 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:46143 every 8 connection(s)
I20260926 04:11:34.691509 18176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:11:34.695257 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18176
I20260926 04:11:34.695363 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:11:34.695559 855 external_mini_cluster.cc:1428] Setting key d52084bb16cda5bf31af10ca25aa8a14
Sep 26 04:11:34 dist-test-slave-xf24 krb5kdc[17801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:34.702657 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.692405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:39513 (local address 127.0.213.254:36463)
0926 04:11:34.692490 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.692493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.692653 (+ 160us) server_negotiation.cc:407] Connection header received
0926 04:11:34.693711 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.693713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.693725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.693790 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:34.694596 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.695131 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.696797 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.697078 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.697134 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.699462 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:34.699463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:34.699464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:34.699471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:34.701054 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.701558 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.701559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.701559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.701619 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.701961 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.701962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.701963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.702097 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:34.702163 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.702508 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.702588 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 04:11:34.702879 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.692307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:36463 (local address 127.0.213.195:39513)
0926 04:11:34.692519 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.692561 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.693573 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.693881 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.693885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.694218 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:34.694484 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.694488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.695260 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.695261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.696683 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.696686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.697158 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.697159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.697284 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.697826 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:34.697835 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:34.699343 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:34.701156 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:34.701163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:34.701165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:34.701418 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:34.701720 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:34.701722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:34.701722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:34.701864 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:34.702188 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:34.702194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:34.702341 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.702467 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.702612 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260926 04:11:34.703495 18305 heartbeater.cc:347] Connected to a master server at 127.0.213.254:36463
I20260926 04:11:34.703622 18305 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:34.703822 18305 heartbeater.cc:511] Master 127.0.213.254:36463 requested a full tablet report, sending...
I20260926 04:11:34.704353 17846 ts_manager.cc:194] Registered new tserver with Master: 336452ef46874f7a9685fb1b39ccc2fa (127.0.213.195:46143)
I20260926 04:11:34.704742 17846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:39513
I20260926 04:11:34.707620 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:11:34.715982 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.709082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.0.213.254:36463)
0926 04:11:34.709161 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.709163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.709258 (+ 95us) server_negotiation.cc:407] Connection header received
0926 04:11:34.709346 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.709347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.709357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.709408 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:34.709988 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.710487 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.711238 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.711451 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.711539 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.713106 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:34.713108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:34.713108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:34.713117 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:34.714675 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.715108 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.715109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.715110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.715180 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:34.715380 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:34.715381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:34.715381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:34.715602 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:34.715683 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.715801 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.715866 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:11:34.717660 17846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:11:34.719380 17846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:11:34.726747 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.722056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:39703 (local address 127.0.0.1:40906)
0926 04:11:34.722530 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.722542 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.722598 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.722952 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.722953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.722955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:34.723005 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.723007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.724370 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.724371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.725510 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.725514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.726382 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.726383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.726508 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.726527 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.726597 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.726653 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":70,"thread_start_us":104,"threads_started":1}
I20260926 04:11:34.726819 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.722565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.0.213.194:39703)
0926 04:11:34.722773 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.722778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.722790 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:11:34.722833 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.722835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.722854 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.722955 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:34.723080 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.724250 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.725618 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.726270 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.726347 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.726411 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.726652 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.726691 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260926 04:11:34.727612 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.722047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43525 (local address 127.0.0.1:43484)
0926 04:11:34.722453 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.722464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.722506 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.723274 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.723275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.723277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:34.723326 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.723327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.724336 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.724337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.725883 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.725886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.727349 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.727350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.727428 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.727433 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.727487 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.727532 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":95,"thread_start_us":114,"threads_started":1}
I20260926 04:11:34.727651 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 1920d004104b40fb85ea4e51768187f5 (DEFAULT_TABLE table=test-table [id=faaeada1022a415ea881476398669401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:34.728035 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1920d004104b40fb85ea4e51768187f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:34.728085 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.722352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.213.193:43525)
0926 04:11:34.723110 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.723115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.723125 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:11:34.723172 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.723173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.723186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.723276 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:34.723396 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.724226 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.726032 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.727199 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.727263 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.727315 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.727925 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.727971 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":84,"threads_started":1}
I20260926 04:11:34.728709 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.722009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:46143 (local address 127.0.0.1:32774)
0926 04:11:34.722204 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.722236 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.722323 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.722719 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.722722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.722724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:34.722802 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.722804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.724386 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.724387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.727063 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.727066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.728251 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.728252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.728499 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.728503 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.728591 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.728636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":143,"thread_start_us":54,"threads_started":1}
I20260926 04:11:34.728730 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.722339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32774 (local address 127.0.213.195:46143)
0926 04:11:34.722513 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.722519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.722530 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:11:34.722614 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.722617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.722635 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.722719 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:34.723036 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.724246 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.727172 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.728037 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.728139 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.728209 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.728607 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.728652 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260926 04:11:34.728875 17971 tablet_service.cc:1467] Processing CreateTablet for tablet 1920d004104b40fb85ea4e51768187f5 (DEFAULT_TABLE table=test-table [id=faaeada1022a415ea881476398669401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:34.729218 17971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1920d004104b40fb85ea4e51768187f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:34.731609 18322 tablet_bootstrap.cc:492] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: Bootstrap starting.
I20260926 04:11:34.732024 18239 tablet_service.cc:1467] Processing CreateTablet for tablet 1920d004104b40fb85ea4e51768187f5 (DEFAULT_TABLE table=test-table [id=faaeada1022a415ea881476398669401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:34.732342 18322 tablet_bootstrap.cc:654] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:34.732417 18239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1920d004104b40fb85ea4e51768187f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:34.732589 18323 tablet_bootstrap.cc:492] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c: Bootstrap starting.
I20260926 04:11:34.733011 18322 log.cc:824] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:34.733410 18323 tablet_bootstrap.cc:654] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:34.733935 18322 tablet_bootstrap.cc:492] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: No bootstrap required, opened a new log
I20260926 04:11:34.734020 18323 log.cc:824] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:34.734021 18322 ts_tablet_manager.cc:1397] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:11:34.734874 18323 tablet_bootstrap.cc:492] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c: No bootstrap required, opened a new log
I20260926 04:11:34.734961 18323 ts_tablet_manager.cc:1397] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:34.735577 18326 tablet_bootstrap.cc:492] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Bootstrap starting.
I20260926 04:11:34.736192 18322 raft_consensus.cc:348] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.736387 18322 raft_consensus.cc:374] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:34.736445 18322 raft_consensus.cc:729] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865fc0e5977c47b382a362c9d2818252, State: Initialized, Role: FOLLOWER
I20260926 04:11:34.736521 18326 tablet_bootstrap.cc:654] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:34.736594 18322 consensus_queue.cc:260] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.736900 18322 ts_tablet_manager.cc:1428] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:34.737032 18171 heartbeater.cc:503] Master 127.0.213.254:36463 was elected leader, sending a full tablet report...
I20260926 04:11:34.737123 18326 log.cc:824] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:34.737084 18323 raft_consensus.cc:348] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.737260 18323 raft_consensus.cc:374] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:34.737300 18323 raft_consensus.cc:729] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fc4b59bae842fbb910451ad283cc5c, State: Initialized, Role: FOLLOWER
I20260926 04:11:34.737448 18323 consensus_queue.cc:260] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.737709 18323 ts_tablet_manager.cc:1428] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:34.737928 18326 tablet_bootstrap.cc:492] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: No bootstrap required, opened a new log
I20260926 04:11:34.737996 18326 ts_tablet_manager.cc:1397] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:34.739459 18326 raft_consensus.cc:348] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.739619 18326 raft_consensus.cc:374] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:34.739665 18326 raft_consensus.cc:729] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336452ef46874f7a9685fb1b39ccc2fa, State: Initialized, Role: FOLLOWER
I20260926 04:11:34.739801 18326 consensus_queue.cc:260] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.740139 18326 ts_tablet_manager.cc:1428] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:34.740159 18037 heartbeater.cc:503] Master 127.0.213.254:36463 was elected leader, sending a full tablet report...
I20260926 04:11:34.740147 18305 heartbeater.cc:503] Master 127.0.213.254:36463 was elected leader, sending a full tablet report...
W20260926 04:11:34.779000 18172 tablet.cc:2369] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:34.792013 18327 raft_consensus.cc:482] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:34.792173 18327 raft_consensus.cc:504] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.792644 18327 leader_election.cc:288] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 336452ef46874f7a9685fb1b39ccc2fa (127.0.213.195:46143), 38fc4b59bae842fbb910451ad283cc5c (127.0.213.193:43525)
I20260926 04:11:34.795650 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.792978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:37235 (local address 127.0.213.193:43525)
0926 04:11:34.793024 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.793026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.793035 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:34.793065 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.793066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.793074 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.793159 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:34.793323 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.793812 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.794799 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.795246 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.795361 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.795371 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.795537 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.795576 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:11:34.795651 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.792859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43525 (local address 127.0.213.194:37235)
0926 04:11:34.792933 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.792946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.792994 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.793162 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.793162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.793164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:34.793246 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.793248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.793899 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.793900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.794696 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.794699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.795367 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.795368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.795451 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.795454 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.795531 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.795572 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":2}
I20260926 04:11:34.795718 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.792958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:51359 (local address 127.0.213.195:46143)
0926 04:11:34.793022 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.793024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.793253 (+ 229us) server_negotiation.cc:407] Connection header received
0926 04:11:34.793317 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.793318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.793328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.793381 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:34.793543 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.794009 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.794945 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.795342 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.795429 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.795437 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.795610 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.795650 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:11:34.795971 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.792880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:46143 (local address 127.0.213.194:51359)
0926 04:11:34.793144 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.793155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:34.793260 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:34.793382 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:34.793382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:34.793384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:34.793464 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.793466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.794098 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.794099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.794843 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:34.794845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.795442 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.795443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:34.795524 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.795527 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:34.795595 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:11:34.795636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260926 04:11:34.796005 17991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1920d004104b40fb85ea4e51768187f5" candidate_uuid: "865fc0e5977c47b382a362c9d2818252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38fc4b59bae842fbb910451ad283cc5c" is_pre_election: true
I20260926 04:11:34.796200 17991 raft_consensus.cc:2454] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865fc0e5977c47b382a362c9d2818252 in term 0.
I20260926 04:11:34.796368 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1920d004104b40fb85ea4e51768187f5" candidate_uuid: "865fc0e5977c47b382a362c9d2818252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336452ef46874f7a9685fb1b39ccc2fa" is_pre_election: true
I20260926 04:11:34.796476 18259 raft_consensus.cc:2454] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865fc0e5977c47b382a362c9d2818252 in term 0.
I20260926 04:11:34.796489 18058 leader_election.cc:302] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38fc4b59bae842fbb910451ad283cc5c, 865fc0e5977c47b382a362c9d2818252; no voters:
I20260926 04:11:34.796610 18327 raft_consensus.cc:2781] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:11:34.796692 18327 raft_consensus.cc:482] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:11:34.796725 18327 raft_consensus.cc:3037] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:34.797410 18327 raft_consensus.cc:504] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.797564 18327 leader_election.cc:288] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [CANDIDATE]: Term 1 election: Requested vote from peers 336452ef46874f7a9685fb1b39ccc2fa (127.0.213.195:46143), 38fc4b59bae842fbb910451ad283cc5c (127.0.213.193:43525)
I20260926 04:11:34.797724 17991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1920d004104b40fb85ea4e51768187f5" candidate_uuid: "865fc0e5977c47b382a362c9d2818252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38fc4b59bae842fbb910451ad283cc5c"
I20260926 04:11:34.797765 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1920d004104b40fb85ea4e51768187f5" candidate_uuid: "865fc0e5977c47b382a362c9d2818252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336452ef46874f7a9685fb1b39ccc2fa"
I20260926 04:11:34.797827 17991 raft_consensus.cc:3037] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:34.797832 18259 raft_consensus.cc:3037] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:34.798758 17991 raft_consensus.cc:2454] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865fc0e5977c47b382a362c9d2818252 in term 1.
I20260926 04:11:34.798888 18259 raft_consensus.cc:2454] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865fc0e5977c47b382a362c9d2818252 in term 1.
I20260926 04:11:34.798981 18058 leader_election.cc:302] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38fc4b59bae842fbb910451ad283cc5c, 865fc0e5977c47b382a362c9d2818252; no voters:
I20260926 04:11:34.799094 18327 raft_consensus.cc:2781] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:34.799268 18327 raft_consensus.cc:686] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [term 1 LEADER]: Becoming Leader. State: Replica: 865fc0e5977c47b382a362c9d2818252, State: Running, Role: LEADER
I20260926 04:11:34.799381 18327 consensus_queue.cc:237] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } }
I20260926 04:11:34.800279 17844 catalog_manager.cc:5495] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 865fc0e5977c47b382a362c9d2818252 (127.0.213.194). New cstate: current_term: 1 leader_uuid: "865fc0e5977c47b382a362c9d2818252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "865fc0e5977c47b382a362c9d2818252" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 39703 } health_report { overall_health: HEALTHY } } }
I20260926 04:11:34.822372 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.819852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.0.213.194:39703)
0926 04:11:34.819991 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:34.819993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:34.820009 (+ 16us) server_negotiation.cc:407] Connection header received
0926 04:11:34.820046 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:34.820047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:34.820099 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:34.820165 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:11:34.820343 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.820834 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.821616 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:34.821804 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:34.821863 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:34.822011 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:11:34.822118 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:11:34.822181 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:34.822217 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 04:11:34.822246 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 04:11:34.824208 18259 raft_consensus.cc:1261] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 1 FOLLOWER]: Refusing update from remote peer 865fc0e5977c47b382a362c9d2818252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:34.824219 17991 raft_consensus.cc:1261] T 1920d004104b40fb85ea4e51768187f5 P 38fc4b59bae842fbb910451ad283cc5c [term 1 FOLLOWER]: Refusing update from remote peer 865fc0e5977c47b382a362c9d2818252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:34.824589 18327 consensus_queue.cc:1035] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38fc4b59bae842fbb910451ad283cc5c" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:34.824746 18332 consensus_queue.cc:1035] T 1920d004104b40fb85ea4e51768187f5 P 865fc0e5977c47b382a362c9d2818252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "336452ef46874f7a9685fb1b39ccc2fa" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:34.827310 18337 mvcc.cc:205] Tried to move back new op lower bound from 7333461585196908544 to 7333461585098919936. Current Snapshot: MvccSnapshot[applied={T|T < 7333461585196908544}]
I20260926 04:11:34.827380 18335 mvcc.cc:205] Tried to move back new op lower bound from 7333461585196908544 to 7333461585098919936. Current Snapshot: MvccSnapshot[applied={T|T < 7333461585196908544}]
I20260926 04:11:34.830753 17844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:34.830879 17844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:34.831730 17844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a9e4c1446c6480dabd2a0530d96b0da: Sending DeleteTablet for 3 replicas of tablet 1920d004104b40fb85ea4e51768187f5
I20260926 04:11:34.832126 17971 tablet_service.cc:1514] Processing DeleteTablet for tablet 1920d004104b40fb85ea4e51768187f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:34 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:43484
I20260926 04:11:34.832129 18239 tablet_service.cc:1514] Processing DeleteTablet for tablet 1920d004104b40fb85ea4e51768187f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:34 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:32774
I20260926 04:11:34.832284 18105 tablet_service.cc:1514] Processing DeleteTablet for tablet 1920d004104b40fb85ea4e51768187f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:34 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:40906
I20260926 04:11:34.832366 18345 tablet_replica.cc:330] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: stopping tablet replica
I20260926 04:11:34.832475 18345 raft_consensus.cc:2229] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:34.832602 18345 raft_consensus.cc:2258] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:34.832636 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 17908
I20260926 04:11:34.835053 18345 ts_tablet_manager.cc:1905] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:34.836956 18345 ts_tablet_manager.cc:1918] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:34.837042 18345 log.cc:1196] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/1920d004104b40fb85ea4e51768187f5
I20260926 04:11:34.837325 18345 ts_tablet_manager.cc:1939] T 1920d004104b40fb85ea4e51768187f5 P 336452ef46874f7a9685fb1b39ccc2fa: Deleting consensus metadata
I20260926 04:11:34.837782 17831 catalog_manager.cc:4841] TS 336452ef46874f7a9685fb1b39ccc2fa (127.0.213.195:46143): tablet 1920d004104b40fb85ea4e51768187f5 (table test-table [id=faaeada1022a415ea881476398669401]) successfully deleted
W20260926 04:11:34.838164 17830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:43525 (error 108)
I20260926 04:11:34.838390 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18042
I20260926 04:11:34.840342 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.840210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43525 (local address 127.0.0.1:43488)
0926 04:11:34.840272 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.840313 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 04:11:34.840467 17830 catalog_manager.cc:4584] TS 38fc4b59bae842fbb910451ad283cc5c (127.0.213.193:43525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1920d004104b40fb85ea4e51768187f5: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43525: connect: Connection refused (error 111)
I20260926 04:11:34.842934 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:34.842804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:39703 (local address 127.0.0.1:40916)
0926 04:11:34.842870 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:34.842902 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.194:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 04:11:34.842940 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18176
I20260926 04:11:34.847800 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 17814
2026-09-26T04:11:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1989 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T04:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:34Z Disabled control of system clock
I20260926 04:11:34.875959 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46571
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:34.960587 18352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:34.960808 18352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:34.960839 18352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:34.961925 18352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:34.961959 18352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:34.961971 18352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:34.961984 18352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:34.961997 18352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:34.963266 18352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:34.963470 18352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:34.963701 18352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:34.968521 18357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.968609 18358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:34.968529 18360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:34.968902 18352 server_base.cc:1034] running on GCE node
I20260926 04:11:34.969185 18352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:34.969498 18352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:34.970670 18352 hybrid_clock.cc:648] HybridClock initialized: now 1790395894970649 us; error 39 us; skew 500 ppm
I20260926 04:11:34.971976 18352 webserver.cc:460] Webserver started at http://127.0.213.254:40981/ using document root <none> and password file <none>
I20260926 04:11:34.972262 18352 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:34.972313 18352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:34.972411 18352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:34.973609 18352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "8dc5bcc61f074daea504a594f3637691"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "aa88ee6f4854bfcaaed41d1288aa4dea"
server_key_iv: "ce48c03058628a21e15b9737b60c66b7"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.973986 18352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "8dc5bcc61f074daea504a594f3637691"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "aa88ee6f4854bfcaaed41d1288aa4dea"
server_key_iv: "ce48c03058628a21e15b9737b60c66b7"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.975432 18352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:34.976303 18366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:34.976528 18352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:34.976652 18352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "8dc5bcc61f074daea504a594f3637691"
format_stamp: "Formatted at 2026-09-26 04:11:34 on dist-test-slave-xf24"
server_key: "aa88ee6f4854bfcaaed41d1288aa4dea"
server_key_iv: "ce48c03058628a21e15b9737b60c66b7"
server_key_version: "encryptionkey@0"
I20260926 04:11:34.976735 18352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:35.001693 18352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:35.002056 18352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:35.002216 18352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:35.006321 18352 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:46571
I20260926 04:11:35.006508 18418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:46571 every 8 connection(s)
I20260926 04:11:35.006814 18352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:35.007865 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:35.010556 18419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691: Bootstrap starting.
I20260926 04:11:35.010777 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18352
I20260926 04:11:35.010885 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:35.011075 855 external_mini_cluster.cc:1428] Setting key 80a2c445627e95e084fe3738a28067c0
I20260926 04:11:35.011379 18419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:35.011775 18419 log.cc:824] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:35.012722 18419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691: No bootstrap required, opened a new log
I20260926 04:11:35.015492 18419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc5bcc61f074daea504a594f3637691" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46571 } }
I20260926 04:11:35.015697 18419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:35.015751 18419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc5bcc61f074daea504a594f3637691, State: Initialized, Role: FOLLOWER
I20260926 04:11:35.015936 18419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc5bcc61f074daea504a594f3637691" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46571 } }
I20260926 04:11:35.016027 18419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:35.016099 18419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:35.016157 18419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:35.017091 18419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc5bcc61f074daea504a594f3637691" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46571 } }
I20260926 04:11:35.017225 18419 leader_election.cc:302] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dc5bcc61f074daea504a594f3637691; no voters:
I20260926 04:11:35.017465 18419 leader_election.cc:288] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:35.017544 18424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:35.017733 18424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [term 1 LEADER]: Becoming Leader. State: Replica: 8dc5bcc61f074daea504a594f3637691, State: Running, Role: LEADER
I20260926 04:11:35.017817 18419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:35.017853 18424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc5bcc61f074daea504a594f3637691" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46571 } }
I20260926 04:11:35.018169 18424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc5bcc61f074daea504a594f3637691. Latest consensus state: current_term: 1 leader_uuid: "8dc5bcc61f074daea504a594f3637691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc5bcc61f074daea504a594f3637691" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46571 } } }
I20260926 04:11:35.018272 18424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:35.018177 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dc5bcc61f074daea504a594f3637691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc5bcc61f074daea504a594f3637691" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46571 } } }
I20260926 04:11:35.018496 18429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:35.018518 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:35.018553 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.011707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.0.213.254:46571)
0926 04:11:35.011965 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.011977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.012020 (+ 43us) server_negotiation.cc:407] Connection header received
0926 04:11:35.012676 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.012687 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.012775 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.012960 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:35.013157 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.014838 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.015732 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.016463 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.016614 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.017385 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:35.017390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:35.017390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:35.017645 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:35.017692 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:35.017710 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:35.018093 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.018148 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.018236 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260926 04:11:35.019013 18429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:35.020699 18429 catalog_manager.cc:1347] Generated new cluster ID: 4b427bae5a31422b8d960c8dcbfebc70
I20260926 04:11:35.020752 18429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:35.030072 18429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:35.030652 18429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:35.036506 18429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691: Generated new TSK 0
I20260926 04:11:35.036727 18429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:35.045567 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46571
--builtin_ntp_servers=127.0.213.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:35.135736 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:35.135991 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:35.136019 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:35.137662 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:35.137728 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:35.137761 18443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:35.139623 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:35.139946 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:35.140420 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:35.147032 18448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:35.147055 18451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:35.147055 18449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:35.147329 18443 server_base.cc:1034] running on GCE node
I20260926 04:11:35.147651 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:35.147946 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:35.149114 18443 hybrid_clock.cc:648] HybridClock initialized: now 1790395895149103 us; error 56 us; skew 500 ppm
I20260926 04:11:35.150569 18443 webserver.cc:460] Webserver started at http://127.0.213.193:35013/ using document root <none> and password file <none>
I20260926 04:11:35.150839 18443 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:35.150907 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:35.151104 18443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:35.152344 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "185fc57a76fc42bda806a292b941f7ae"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "a33b39ae02f280472a336e550c4d85bb"
server_key_iv: "dff97891300311d9c84d14b19fd17352"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.152710 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "185fc57a76fc42bda806a292b941f7ae"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "a33b39ae02f280472a336e550c4d85bb"
server_key_iv: "dff97891300311d9c84d14b19fd17352"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.154173 18443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:11:35.154989 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.155194 18443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:35.155293 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "185fc57a76fc42bda806a292b941f7ae"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "a33b39ae02f280472a336e550c4d85bb"
server_key_iv: "dff97891300311d9c84d14b19fd17352"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.155387 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:35.167860 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:35.168294 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:35.168471 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:35.168715 18443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:35.169081 18443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:35.169143 18443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.169196 18443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:35.169229 18443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.175690 18443 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:36481
I20260926 04:11:35.175769 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:36481 every 8 connection(s)
I20260926 04:11:35.176230 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:11:35.181186 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18443
I20260926 04:11:35.181303 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:11:35.181516 855 external_mini_cluster.cc:1428] Setting key 8911138428d8aa6d0019447f2667af91
I20260926 04:11:35.182698 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46571
--builtin_ntp_servers=127.0.213.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:11:35.182782 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.177150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:43757 (local address 127.0.213.254:46571)
0926 04:11:35.177233 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.177235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.177516 (+ 281us) server_negotiation.cc:407] Connection header received
0926 04:11:35.178094 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.178096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.178108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.178186 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:35.179000 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.179776 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.180894 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.181194 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.181285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.182470 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:35.182472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:35.182472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:35.182478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:35.182497 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:35.182502 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:35.182575 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.182658 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.182686 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 04:11:35.183014 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.177105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46571 (local address 127.0.213.193:43757)
0926 04:11:35.177344 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.177394 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.177981 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.178259 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.178263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.178272 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:35.178857 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.178861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.179876 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.179878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.180815 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.180818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.181330 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.181331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.181520 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.182298 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:35.182299 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:35.182338 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:35.182374 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:35.182573 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:35.182578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:35.182581 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.182637 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.182724 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260926 04:11:35.183722 18571 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46571
I20260926 04:11:35.183847 18571 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:35.184109 18571 heartbeater.cc:511] Master 127.0.213.254:46571 requested a full tablet report, sending...
I20260926 04:11:35.184708 18383 ts_manager.cc:194] Registered new tserver with Master: 185fc57a76fc42bda806a292b941f7ae (127.0.213.193:36481)
I20260926 04:11:35.185331 18383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.193:43757
W20260926 04:11:35.265213 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:35.265439 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:35.265468 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:35.266717 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:35.266767 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:35.266794 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:35.268383 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:35.268657 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:35.268971 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:35.274405 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:35.274415 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:35.274415 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:35.274699 18574 server_base.cc:1034] running on GCE node
I20260926 04:11:35.274933 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:35.275167 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:35.276352 18574 hybrid_clock.cc:648] HybridClock initialized: now 1790395895276337 us; error 30 us; skew 500 ppm
I20260926 04:11:35.278080 18574 webserver.cc:460] Webserver started at http://127.0.213.194:37809/ using document root <none> and password file <none>
I20260926 04:11:35.278319 18574 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:35.278363 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:35.278474 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:35.279582 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "6b669f605ab34b0eabccbe42729ddb4f"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "5ebb6af572c3b5c289460a5bfd2206de"
server_key_iv: "ca71a15f2c6ccfd923dd791fbcfaff11"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.280179 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "6b669f605ab34b0eabccbe42729ddb4f"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "5ebb6af572c3b5c289460a5bfd2206de"
server_key_iv: "ca71a15f2c6ccfd923dd791fbcfaff11"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.281563 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:35.282356 18588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.282549 18574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:35.282668 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "6b669f605ab34b0eabccbe42729ddb4f"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "5ebb6af572c3b5c289460a5bfd2206de"
server_key_iv: "ca71a15f2c6ccfd923dd791fbcfaff11"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.282759 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:35.301066 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:35.301455 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:35.301649 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:35.301925 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:35.302310 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:35.302372 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.302418 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:35.302451 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.308861 18574 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:33937
I20260926 04:11:35.308902 18701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:33937 every 8 connection(s)
I20260926 04:11:35.309300 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:35.314738 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.310108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:43623 (local address 127.0.213.254:46571)
0926 04:11:35.310178 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.310180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.310416 (+ 236us) server_negotiation.cc:407] Connection header received
0926 04:11:35.310935 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.310936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.310947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.311017 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:35.311801 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.312375 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.313431 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.313626 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.313683 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.314498 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:35.314499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:35.314500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:35.314504 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:35.314518 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:35.314522 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:35.314566 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.314654 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.314676 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:11:35.314999 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.310038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46571 (local address 127.0.213.194:43623)
0926 04:11:35.310281 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.310318 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.310837 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.311077 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.311082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.311091 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:35.311709 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.311715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.312462 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.312463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.313339 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.313341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.313711 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.313712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.313830 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.314350 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:35.314351 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:35.314380 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:35.314404 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:35.314581 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:35.314585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:35.314587 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.314642 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.314725 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260926 04:11:35.315564 18702 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46571
I20260926 04:11:35.315670 18702 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:35.315850 18702 heartbeater.cc:511] Master 127.0.213.254:46571 requested a full tablet report, sending...
I20260926 04:11:35.316260 18383 ts_manager.cc:194] Registered new tserver with Master: 6b669f605ab34b0eabccbe42729ddb4f (127.0.213.194:33937)
I20260926 04:11:35.316632 18383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.194:43623
I20260926 04:11:35.318351 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18574
I20260926 04:11:35.318475 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:11:35.318766 855 external_mini_cluster.cc:1428] Setting key 749140df58e99fe8a36c2071d7082cf4
I20260926 04:11:35.319868 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46571
--builtin_ntp_servers=127.0.213.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:35.401656 18705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:35.401875 18705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:35.401898 18705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:35.403127 18705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:35.403177 18705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:35.403211 18705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:35.404664 18705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:35.404934 18705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:35.405279 18705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:35.410524 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:35.410594 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:35.410604 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:35.410820 18705 server_base.cc:1034] running on GCE node
I20260926 04:11:35.411108 18705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:35.411356 18705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:35.412609 18705 hybrid_clock.cc:648] HybridClock initialized: now 1790395895412600 us; error 42 us; skew 500 ppm
I20260926 04:11:35.414201 18705 webserver.cc:460] Webserver started at http://127.0.213.195:43213/ using document root <none> and password file <none>
I20260926 04:11:35.414564 18705 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:35.414649 18705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:35.414819 18705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:35.416023 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "4f1daabefed347138e4e6f5c652a147a"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "c293b2e13e868c201170f6d0c78be70b"
server_key_iv: "ff17804982780525561a33e44aa3d8d9"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.416500 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "4f1daabefed347138e4e6f5c652a147a"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "c293b2e13e868c201170f6d0c78be70b"
server_key_iv: "ff17804982780525561a33e44aa3d8d9"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.418079 18705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:11:35.419088 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.419296 18705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:35.419406 18705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "4f1daabefed347138e4e6f5c652a147a"
format_stamp: "Formatted at 2026-09-26 04:11:35 on dist-test-slave-xf24"
server_key: "c293b2e13e868c201170f6d0c78be70b"
server_key_iv: "ff17804982780525561a33e44aa3d8d9"
server_key_version: "encryptionkey@0"
I20260926 04:11:35.419555 18705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:35.460878 18705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:35.461231 18705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:35.461395 18705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:35.461658 18705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:35.462023 18705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:35.462086 18705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.462136 18705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:35.462172 18705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:35.468487 18705 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:38897
I20260926 04:11:35.468564 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:38897 every 8 connection(s)
I20260926 04:11:35.468943 18705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:11:35.474561 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.469773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:48499 (local address 127.0.213.254:46571)
0926 04:11:35.469843 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.469846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.470132 (+ 286us) server_negotiation.cc:407] Connection header received
0926 04:11:35.470623 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.470625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.470663 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.470752 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:35.471483 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.472029 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.473109 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.473319 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.473390 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.474291 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:35.474292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:35.474292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:35.474297 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:35.474313 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:35.474317 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:35.474371 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.474474 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.474499 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:11:35.474706 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.469764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46571 (local address 127.0.213.195:48499)
0926 04:11:35.469959 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.470009 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.470508 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.470825 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.470828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.470837 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:35.471392 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.471396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.472188 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.472189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.473005 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.473007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.473409 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.473410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.473535 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.474139 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:35.474139 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:35.474168 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:35.474191 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:35.474377 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:35.474380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:35.474383 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.474433 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.474517 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260926 04:11:35.475214 18833 heartbeater.cc:347] Connected to a master server at 127.0.213.254:46571
I20260926 04:11:35.475327 18833 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:35.475543 18833 heartbeater.cc:511] Master 127.0.213.254:46571 requested a full tablet report, sending...
I20260926 04:11:35.475560 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18705
I20260926 04:11:35.475644 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:11:35.475858 855 external_mini_cluster.cc:1428] Setting key e8b998cb14aca60a3b5adcfaeda1cd21
I20260926 04:11:35.476022 18383 ts_manager.cc:194] Registered new tserver with Master: 4f1daabefed347138e4e6f5c652a147a (127.0.213.195:38897)
I20260926 04:11:35.476586 18383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.195:48499
I20260926 04:11:35.476771 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:11:35.481206 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.478359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.0.213.254:46571)
0926 04:11:35.478411 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.478413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.478521 (+ 108us) server_negotiation.cc:407] Connection header received
0926 04:11:35.478562 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.478562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.478571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.478628 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:35.478761 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.479302 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.480125 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.480321 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.480402 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.481004 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:35.481006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:35.481006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:35.481010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:35.481022 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:35.481026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:35.481080 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.481129 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.481149 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 04:11:35.483103 18383 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:11:35.484802 18383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:11:35.491665 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.487174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:38897 (local address 127.0.0.1:51978)
0926 04:11:35.487351 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.487396 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.487471 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.487871 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.487874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.487877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:35.487998 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.488001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.489370 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.489370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.490661 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.490664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.491358 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.491359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.491445 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.491458 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.491531 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.491586 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":98,"thread_start_us":66,"threads_started":1}
I20260926 04:11:35.491678 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.487524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.0.213.195:38897)
0926 04:11:35.487722 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.487726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.487734 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:11:35.487767 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.487769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.487780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.487842 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:35.488113 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.489294 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.490767 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.491265 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.491358 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.491456 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.491544 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.491585 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260926 04:11:35.492033 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.487228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36481 (local address 127.0.0.1:55940)
0926 04:11:35.487524 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.487535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.487590 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.487924 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.487925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.487928 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:35.487991 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.487993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.489420 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.489421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.490457 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.490459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.491789 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.491790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.491853 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.491857 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.491913 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.491957 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":37,"thread_start_us":73,"threads_started":1}
I20260926 04:11:35.492177 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.487500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.0.213.194:33937)
0926 04:11:35.487688 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.487693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.488164 (+ 471us) server_negotiation.cc:407] Connection header received
0926 04:11:35.488214 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.488216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.488232 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.488305 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:35.488506 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.489208 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.490753 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.491477 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.491548 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.491600 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.492006 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.492082 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260926 04:11:35.492295 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.487321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.0.213.193:36481)
0926 04:11:35.487718 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.487722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.487731 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:35.487777 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.487778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.487790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.487895 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:35.488173 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.489323 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.490547 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.491263 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.491366 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.491457 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.492167 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.492216 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":105,"threads_started":1}
I20260926 04:11:35.492334 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.487396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:33937 (local address 127.0.0.1:33662)
0926 04:11:35.487981 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.487992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.488041 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.488374 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.488375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.488376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:35.488418 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.488419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.489305 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.489306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.490661 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.490664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.491805 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.491806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.491853 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.491860 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.491924 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.491958 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":41,"thread_start_us":58,"threads_started":1}
I20260926 04:11:35.492761 18767 tablet_service.cc:1467] Processing CreateTablet for tablet d2b5c82dacea45178aea8f08d6cdbb2b (DEFAULT_TABLE table=test-table [id=95a1498c7bb24dab9458bafa4fd23843]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:35.492982 18505 tablet_service.cc:1467] Processing CreateTablet for tablet d2b5c82dacea45178aea8f08d6cdbb2b (DEFAULT_TABLE table=test-table [id=95a1498c7bb24dab9458bafa4fd23843]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:35.493175 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b5c82dacea45178aea8f08d6cdbb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:35.493122 18636 tablet_service.cc:1467] Processing CreateTablet for tablet d2b5c82dacea45178aea8f08d6cdbb2b (DEFAULT_TABLE table=test-table [id=95a1498c7bb24dab9458bafa4fd23843]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:35.493337 18505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b5c82dacea45178aea8f08d6cdbb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:35.493453 18636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b5c82dacea45178aea8f08d6cdbb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:35.495952 18847 tablet_bootstrap.cc:492] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Bootstrap starting.
I20260926 04:11:35.496789 18847 tablet_bootstrap.cc:654] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:35.497081 18849 tablet_bootstrap.cc:492] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: Bootstrap starting.
I20260926 04:11:35.497321 18847 log.cc:824] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:35.497750 18849 tablet_bootstrap.cc:654] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:35.498077 18849 log.cc:824] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:35.498208 18847 tablet_bootstrap.cc:492] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: No bootstrap required, opened a new log
I20260926 04:11:35.498281 18847 ts_tablet_manager.cc:1397] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:35.498301 18848 tablet_bootstrap.cc:492] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae: Bootstrap starting.
I20260926 04:11:35.499028 18848 tablet_bootstrap.cc:654] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:35.499645 18848 log.cc:824] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:35.500443 18848 tablet_bootstrap.cc:492] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae: No bootstrap required, opened a new log
I20260926 04:11:35.500528 18848 ts_tablet_manager.cc:1397] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 04:11:35.500864 18849 tablet_bootstrap.cc:492] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: No bootstrap required, opened a new log
I20260926 04:11:35.500982 18849 ts_tablet_manager.cc:1397] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 04:11:35.501302 18847 raft_consensus.cc:348] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.501467 18847 raft_consensus.cc:374] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:35.501510 18847 raft_consensus.cc:729] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1daabefed347138e4e6f5c652a147a, State: Initialized, Role: FOLLOWER
I20260926 04:11:35.501667 18847 consensus_queue.cc:260] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.501950 18847 ts_tablet_manager.cc:1428] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 04:11:35.501963 18833 heartbeater.cc:503] Master 127.0.213.254:46571 was elected leader, sending a full tablet report...
I20260926 04:11:35.503365 18848 raft_consensus.cc:348] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.503554 18848 raft_consensus.cc:374] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:35.503594 18848 raft_consensus.cc:729] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185fc57a76fc42bda806a292b941f7ae, State: Initialized, Role: FOLLOWER
I20260926 04:11:35.503638 18849 raft_consensus.cc:348] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.503722 18848 consensus_queue.cc:260] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.503795 18849 raft_consensus.cc:374] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:35.503839 18849 raft_consensus.cc:729] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b669f605ab34b0eabccbe42729ddb4f, State: Initialized, Role: FOLLOWER
I20260926 04:11:35.503957 18849 consensus_queue.cc:260] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.504082 18848 ts_tablet_manager.cc:1428] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 04:11:35.504158 18571 heartbeater.cc:503] Master 127.0.213.254:46571 was elected leader, sending a full tablet report...
I20260926 04:11:35.504258 18849 ts_tablet_manager.cc:1428] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:35.504314 18702 heartbeater.cc:503] Master 127.0.213.254:46571 was elected leader, sending a full tablet report...
I20260926 04:11:35.529381 18855 raft_consensus.cc:482] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:35.529557 18855 raft_consensus.cc:504] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.530018 18855 leader_election.cc:288] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f1daabefed347138e4e6f5c652a147a (127.0.213.195:38897), 185fc57a76fc42bda806a292b941f7ae (127.0.213.193:36481)
I20260926 04:11:35.533226 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.530396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:52259 (local address 127.0.213.193:36481)
0926 04:11:35.530467 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.530469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.530478 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:35.530509 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.530510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.530523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.530580 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:35.530796 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.531290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.532412 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.532899 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.532966 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.532974 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.533132 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.533167 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 04:11:35.533254 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.530239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36481 (local address 127.0.213.194:52259)
0926 04:11:35.530316 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.530328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.530372 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.530605 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.530606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.530614 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:35.530703 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.530705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.531386 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.531387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.532317 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.532319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.532976 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.532977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.533053 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.533057 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.533121 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.533167 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 04:11:35.533619 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.530392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:49551 (local address 127.0.213.195:38897)
0926 04:11:35.530444 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.530446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.530551 (+ 105us) server_negotiation.cc:407] Connection header received
0926 04:11:35.530606 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.530607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.530617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.530692 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:35.530895 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.531417 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.532722 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.533232 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.533302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.533309 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.533504 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.533554 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:11:35.533633 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.530251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:38897 (local address 127.0.213.194:49551)
0926 04:11:35.530475 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.530486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:35.530542 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:35.530711 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:35.530712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:35.530713 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:35.530800 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.530802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.531657 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.531658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.532629 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:35.532631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.533326 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.533327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:35.533419 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.533423 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:35.533495 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:11:35.533533 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260926 04:11:35.533723 18525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b5c82dacea45178aea8f08d6cdbb2b" candidate_uuid: "6b669f605ab34b0eabccbe42729ddb4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185fc57a76fc42bda806a292b941f7ae" is_pre_election: true
I20260926 04:11:35.533893 18525 raft_consensus.cc:2454] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b669f605ab34b0eabccbe42729ddb4f in term 0.
I20260926 04:11:35.533995 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b5c82dacea45178aea8f08d6cdbb2b" candidate_uuid: "6b669f605ab34b0eabccbe42729ddb4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f1daabefed347138e4e6f5c652a147a" is_pre_election: true
I20260926 04:11:35.534188 18787 raft_consensus.cc:2454] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b669f605ab34b0eabccbe42729ddb4f in term 0.
I20260926 04:11:35.534180 18590 leader_election.cc:302] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185fc57a76fc42bda806a292b941f7ae, 6b669f605ab34b0eabccbe42729ddb4f; no voters:
I20260926 04:11:35.534354 18855 raft_consensus.cc:2781] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:11:35.534439 18855 raft_consensus.cc:482] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:11:35.534480 18855 raft_consensus.cc:3037] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:35.535212 18855 raft_consensus.cc:504] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.535379 18855 leader_election.cc:288] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [CANDIDATE]: Term 1 election: Requested vote from peers 4f1daabefed347138e4e6f5c652a147a (127.0.213.195:38897), 185fc57a76fc42bda806a292b941f7ae (127.0.213.193:36481)
I20260926 04:11:35.535542 18525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b5c82dacea45178aea8f08d6cdbb2b" candidate_uuid: "6b669f605ab34b0eabccbe42729ddb4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185fc57a76fc42bda806a292b941f7ae"
I20260926 04:11:35.535601 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b5c82dacea45178aea8f08d6cdbb2b" candidate_uuid: "6b669f605ab34b0eabccbe42729ddb4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f1daabefed347138e4e6f5c652a147a"
I20260926 04:11:35.535673 18787 raft_consensus.cc:3037] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:35.535655 18525 raft_consensus.cc:3037] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:35.536651 18787 raft_consensus.cc:2454] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b669f605ab34b0eabccbe42729ddb4f in term 1.
I20260926 04:11:35.536650 18525 raft_consensus.cc:2454] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b669f605ab34b0eabccbe42729ddb4f in term 1.
I20260926 04:11:35.536849 18589 leader_election.cc:302] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f1daabefed347138e4e6f5c652a147a, 6b669f605ab34b0eabccbe42729ddb4f; no voters:
I20260926 04:11:35.537026 18855 raft_consensus.cc:2781] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:35.537206 18855 raft_consensus.cc:686] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 1 LEADER]: Becoming Leader. State: Replica: 6b669f605ab34b0eabccbe42729ddb4f, State: Running, Role: LEADER
I20260926 04:11:35.537317 18855 consensus_queue.cc:237] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } }
I20260926 04:11:35.538304 18382 catalog_manager.cc:5495] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b669f605ab34b0eabccbe42729ddb4f (127.0.213.194). New cstate: current_term: 1 leader_uuid: "6b669f605ab34b0eabccbe42729ddb4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b669f605ab34b0eabccbe42729ddb4f" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 33937 } health_report { overall_health: HEALTHY } } }
I20260926 04:11:35.554553 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.551925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.213.194:33937)
0926 04:11:35.551990 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:35.551992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:35.552004 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:11:35.552120 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:35.552121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:35.552150 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:35.552262 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:11:35.552405 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.552908 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.553684 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:35.553886 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:35.553965 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:35.554152 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:11:35.554254 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:11:35.554342 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:35.554380 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 04:11:35.554402 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:11:35.556519 18525 raft_consensus.cc:1261] T d2b5c82dacea45178aea8f08d6cdbb2b P 185fc57a76fc42bda806a292b941f7ae [term 1 FOLLOWER]: Refusing update from remote peer 6b669f605ab34b0eabccbe42729ddb4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:35.556543 18787 raft_consensus.cc:1261] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 1 FOLLOWER]: Refusing update from remote peer 6b669f605ab34b0eabccbe42729ddb4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:35.556927 18857 consensus_queue.cc:1035] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "185fc57a76fc42bda806a292b941f7ae" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:35.557070 18855 consensus_queue.cc:1035] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f1daabefed347138e4e6f5c652a147a" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:35.560268 18861 mvcc.cc:205] Tried to move back new op lower bound from 7333461588196614144 to 7333461588121817088. Current Snapshot: MvccSnapshot[applied={T|T < 7333461588196614144}]
I20260926 04:11:35.562485 18383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:35.562611 18383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:35.563372 18383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dc5bcc61f074daea504a594f3637691: Sending DeleteTablet for 3 replicas of tablet d2b5c82dacea45178aea8f08d6cdbb2b
I20260926 04:11:35.563750 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b5c82dacea45178aea8f08d6cdbb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:35 UTC) from {username='slave'} at 127.0.0.1:51978
I20260926 04:11:35.563768 18636 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b5c82dacea45178aea8f08d6cdbb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:35 UTC) from {username='slave'} at 127.0.0.1:33662
I20260926 04:11:35.563787 18505 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b5c82dacea45178aea8f08d6cdbb2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:35 UTC) from {username='slave'} at 127.0.0.1:55940
I20260926 04:11:35.564216 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18443
I20260926 04:11:35.568224 18871 tablet_replica.cc:330] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: stopping tablet replica
I20260926 04:11:35.568372 18871 raft_consensus.cc:2229] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:35.568514 18871 raft_consensus.cc:2258] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:35.569046 18871 ts_tablet_manager.cc:1905] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:35.570868 18871 ts_tablet_manager.cc:1918] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:35.570947 18871 log.cc:1196] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/d2b5c82dacea45178aea8f08d6cdbb2b
I20260926 04:11:35.571215 18871 ts_tablet_manager.cc:1939] T d2b5c82dacea45178aea8f08d6cdbb2b P 4f1daabefed347138e4e6f5c652a147a: Deleting consensus metadata
I20260926 04:11:35.571719 18367 catalog_manager.cc:4841] TS 4f1daabefed347138e4e6f5c652a147a (127.0.213.195:38897): tablet d2b5c82dacea45178aea8f08d6cdbb2b (table test-table [id=95a1498c7bb24dab9458bafa4fd23843]) successfully deleted
W20260926 04:11:35.572042 18368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:36481 (error 108)
I20260926 04:11:35.572202 18869 tablet_replica.cc:330] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f: stopping tablet replica
I20260926 04:11:35.572345 18869 raft_consensus.cc:2229] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:11:35.572463 18869 raft_consensus.cc:2258] T d2b5c82dacea45178aea8f08d6cdbb2b P 6b669f605ab34b0eabccbe42729ddb4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:35.572585 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.572428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36481 (local address 127.0.0.1:55952)
0926 04:11:35.572480 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.572537 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:36481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 04:11:35.572674 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18574
W20260926 04:11:35.572703 18368 catalog_manager.cc:4584] TS 185fc57a76fc42bda806a292b941f7ae (127.0.213.193:36481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2b5c82dacea45178aea8f08d6cdbb2b: Network error: Client connection negotiation failed: client connection to 127.0.213.193:36481: connect: Connection refused (error 111)
I20260926 04:11:35.577672 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18705
I20260926 04:11:35.577783 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:35.577657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:33937 (local address 127.0.0.1:33680)
0926 04:11:35.577703 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:35.577738 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.194:33937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 04:11:35.582278 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18352
2026-09-26T04:11:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (733 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:35 dist-test-slave-xf24 krb5kdc[18875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:35 dist-test-slave-xf24 krb5kdc[18875](info): set up 2 sockets
Sep 26 04:11:35 dist-test-slave-xf24 krb5kdc[18875](info): commencing operation
krb5kdc: starting...
W20260926 04:11:36.647909 855 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:36 dist-test-slave-xf24 krb5kdc[18875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:11:36.775978 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40005
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:40005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:36.862138 18888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:36.862346 18888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:36.862370 18888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:36.863453 18888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:36.863487 18888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:36.863500 18888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:36.863513 18888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:36.863525 18888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:36.865006 18888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:40005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:40005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:36.865197 18888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:36.865360 18888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:36.870293 18894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:36.870293 18893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:36.870489 18888 server_base.cc:1034] running on GCE node
W20260926 04:11:36.870267 18896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:36.870813 18888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:36.871153 18888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:36.872303 18888 hybrid_clock.cc:648] HybridClock initialized: now 1790395896872276 us; error 40 us; skew 500 ppm
Sep 26 04:11:36 dist-test-slave-xf24 krb5kdc[18875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:36.881034 18888 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:11:36.881553 18888 webserver.cc:460] Webserver started at http://127.0.213.254:38621/ using document root <none> and password file <none>
I20260926 04:11:36.881799 18888 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:36.881868 18888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:36.882018 18888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:36.883162 18888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "fda494e489844b46a77e5a64b98f6710"
format_stamp: "Formatted at 2026-09-26 04:11:36 on dist-test-slave-xf24"
server_key: "5c6e178dbbaf21c4c4c48a2210876cef"
server_key_iv: "19d278a47e6a4411b4ce2bb405e0aeff"
server_key_version: "encryptionkey@0"
I20260926 04:11:36.883549 18888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "fda494e489844b46a77e5a64b98f6710"
format_stamp: "Formatted at 2026-09-26 04:11:36 on dist-test-slave-xf24"
server_key: "5c6e178dbbaf21c4c4c48a2210876cef"
server_key_iv: "19d278a47e6a4411b4ce2bb405e0aeff"
server_key_version: "encryptionkey@0"
I20260926 04:11:36.885159 18888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:11:36.886132 18903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:36.886468 18888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:36.886611 18888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "fda494e489844b46a77e5a64b98f6710"
format_stamp: "Formatted at 2026-09-26 04:11:36 on dist-test-slave-xf24"
server_key: "5c6e178dbbaf21c4c4c48a2210876cef"
server_key_iv: "19d278a47e6a4411b4ce2bb405e0aeff"
server_key_version: "encryptionkey@0"
I20260926 04:11:36.886703 18888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:36.897106 18888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:36.899511 18888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:36.899722 18888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:36.904029 18888 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:40005
I20260926 04:11:36.904114 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:40005 every 8 connection(s)
I20260926 04:11:36.904522 18888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:36.905380 18956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:36.907625 18956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710: Bootstrap starting.
I20260926 04:11:36.908257 18956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:36.908501 18956 log.cc:824] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:36.909169 18956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710: No bootstrap required, opened a new log
I20260926 04:11:36.910696 18956 raft_consensus.cc:348] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda494e489844b46a77e5a64b98f6710" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40005 } }
I20260926 04:11:36.910840 18956 raft_consensus.cc:374] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:36.910863 18956 raft_consensus.cc:729] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda494e489844b46a77e5a64b98f6710, State: Initialized, Role: FOLLOWER
I20260926 04:11:36.910969 18956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda494e489844b46a77e5a64b98f6710" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40005 } }
I20260926 04:11:36.911051 18956 raft_consensus.cc:388] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:36.911088 18956 raft_consensus.cc:482] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:36.911142 18956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:36.911688 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18888
I20260926 04:11:36.911794 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:36.911803 18956 raft_consensus.cc:504] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda494e489844b46a77e5a64b98f6710" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40005 } }
I20260926 04:11:36.911908 18956 leader_election.cc:302] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda494e489844b46a77e5a64b98f6710; no voters:
I20260926 04:11:36.912011 855 external_mini_cluster.cc:1428] Setting key 76443da791850beeeeeea0083aad46c5
I20260926 04:11:36.912109 18956 leader_election.cc:288] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:36.912171 18959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:36.913514 18959 raft_consensus.cc:686] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [term 1 LEADER]: Becoming Leader. State: Replica: fda494e489844b46a77e5a64b98f6710, State: Running, Role: LEADER
I20260926 04:11:36.913606 18956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:36.913623 18959 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda494e489844b46a77e5a64b98f6710" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40005 } }
I20260926 04:11:36.914024 18959 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda494e489844b46a77e5a64b98f6710. Latest consensus state: current_term: 1 leader_uuid: "fda494e489844b46a77e5a64b98f6710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda494e489844b46a77e5a64b98f6710" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40005 } } }
I20260926 04:11:36.914112 18959 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:36.914250 18961 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fda494e489844b46a77e5a64b98f6710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda494e489844b46a77e5a64b98f6710" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 40005 } } }
I20260926 04:11:36.914323 18961 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:36.914470 18968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:36.915028 18968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 04:11:36 dist-test-slave-xf24 krb5kdc[18875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:36.921182 18968 catalog_manager.cc:1347] Generated new cluster ID: 07bcb9ce7aac416899b0f63f4d7c1716
I20260926 04:11:36.921247 18968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:36.925434 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:36.913431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.213.254:40005)
0926 04:11:36.915037 (+ 1606us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:36.915045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:36.915098 (+ 53us) server_negotiation.cc:407] Connection header received
0926 04:11:36.915233 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:36.915241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:36.915301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:36.915461 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:36.916242 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:36.917148 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:36.917940 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:36.918386 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:36.918530 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:36.920984 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:36.920988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:36.920989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:36.921001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:36.923052 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:36.923470 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:36.923473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:36.923473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:36.923582 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:36.923876 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:36.923877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:36.923877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:36.924275 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:36.924890 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:36.924973 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 04:11:36.925137 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":62,"threads_started":1}
I20260926 04:11:36.943215 18968 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:36.944048 18968 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:36.957140 18968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710: Generated new TSK 0
I20260926 04:11:36.957381 18968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:11:36.988029 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40005
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:37.072178 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:37.072395 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:37.072420 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:37.073719 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:37.073773 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:37.073802 18982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:37.075351 18982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:37.075559 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:37.075754 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:11:37.081300 18982 server_base.cc:1034] running on GCE node
W20260926 04:11:37.081264 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.081272 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.081288 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:37.081841 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:37.082082 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:37.083263 18982 hybrid_clock.cc:648] HybridClock initialized: now 1790395897083250 us; error 31 us; skew 500 ppm
Sep 26 04:11:37 dist-test-slave-xf24 krb5kdc[18875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:37.086336 18982 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:11:37.086892 18982 webserver.cc:460] Webserver started at http://127.0.213.193:33537/ using document root <none> and password file <none>
I20260926 04:11:37.087136 18982 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:37.087206 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:37.087369 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:37.088583 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "a89033d2a5ab4993b34935ed938bdd63"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "56a1a49816eae9da64f4e6ff7d387831"
server_key_iv: "c17778679af9ae33bef1fe5a4fd2fc68"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.088964 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "a89033d2a5ab4993b34935ed938bdd63"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "56a1a49816eae9da64f4e6ff7d387831"
server_key_iv: "c17778679af9ae33bef1fe5a4fd2fc68"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.090512 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 04:11:37.091403 18997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.091794 18982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:37.091921 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "a89033d2a5ab4993b34935ed938bdd63"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "56a1a49816eae9da64f4e6ff7d387831"
server_key_iv: "c17778679af9ae33bef1fe5a4fd2fc68"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.092005 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:37.105252 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:37.107707 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:37.107908 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:37.108255 18982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:37.108609 18982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:37.108667 18982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.108711 18982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:37.108737 18982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.116266 18982 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:37401
I20260926 04:11:37.116318 19110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:37401 every 8 connection(s)
I20260926 04:11:37.116693 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
Sep 26 04:11:37 dist-test-slave-xf24 krb5kdc[18875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:37.123793 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 18982
I20260926 04:11:37.123931 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:11:37.124255 855 external_mini_cluster.cc:1428] Setting key 7c8b8eb23cc0c3f04edeccd55712521b
I20260926 04:11:37.127905 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.117593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:37793 (local address 127.0.213.254:40005)
0926 04:11:37.117681 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.117684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.117792 (+ 108us) server_negotiation.cc:407] Connection header received
0926 04:11:37.118818 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.118820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.118834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.118897 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:37.119569 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.120169 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.121237 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.121447 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.121515 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.124045 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:37.124048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:37.124048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:37.124088 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:37.126014 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.126721 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.126723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.126723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.126803 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.127091 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.127092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.127093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.127227 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:37.127305 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.127690 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.127808 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 04:11:37.128234 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.117488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40005 (local address 127.0.213.193:37793)
0926 04:11:37.117650 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.117698 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.118681 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.118956 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.118959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.119255 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:37.119454 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.119458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.120279 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.120280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.121131 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.121133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.121581 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.121583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.121704 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.122343 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:37.122352 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:37.123936 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:37.126123 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:37.126131 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:37.126133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:37.126593 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:37.126877 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:37.126878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:37.126878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:37.127012 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:37.127308 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:37.127315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:37.127467 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.127661 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.127853 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260926 04:11:37.128860 19111 heartbeater.cc:347] Connected to a master server at 127.0.213.254:40005
I20260926 04:11:37.128978 19111 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:37.129287 19111 heartbeater.cc:511] Master 127.0.213.254:40005 requested a full tablet report, sending...
I20260926 04:11:37.129966 18920 ts_manager.cc:194] Registered new tserver with Master: a89033d2a5ab4993b34935ed938bdd63 (127.0.213.193:37401)
I20260926 04:11:37.130992 18920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:37793
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:11:37.156448 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40005
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:37.241098 19116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:37.241359 19116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:37.241400 19116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:37.242699 19116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:37.242767 19116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:37.242815 19116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:37.244463 19116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:37.244758 19116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:37.245010 19116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:37.250347 19121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.250373 19122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.250347 19124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:37.250630 19116 server_base.cc:1034] running on GCE node
I20260926 04:11:37.250890 19116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:37.251169 19116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:37.252355 19116 hybrid_clock.cc:648] HybridClock initialized: now 1790395897252325 us; error 46 us; skew 500 ppm
Sep 26 04:11:37 dist-test-slave-xf24 krb5kdc[18875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:37.255285 19116 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:11:37.255824 19116 webserver.cc:460] Webserver started at http://127.0.213.194:36865/ using document root <none> and password file <none>
I20260926 04:11:37.256173 19116 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:37.256249 19116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:37.256398 19116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:37.257546 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "bcf2863e70c14fab83669368754489df"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "77749f4833418a4942f4539310e1289b"
server_key_iv: "143b5128df6323c402d019925a803bca"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.257925 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "bcf2863e70c14fab83669368754489df"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "77749f4833418a4942f4539310e1289b"
server_key_iv: "143b5128df6323c402d019925a803bca"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.259361 19116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:37.260278 19131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.260550 19116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:37.260670 19116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "bcf2863e70c14fab83669368754489df"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "77749f4833418a4942f4539310e1289b"
server_key_iv: "143b5128df6323c402d019925a803bca"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.260754 19116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:37.271349 19116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:37.273499 19116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:37.273674 19116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:37.273931 19116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:37.274262 19116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:37.274322 19116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.274365 19116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:37.274391 19116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.280525 19116 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:40755
I20260926 04:11:37.280602 19244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:40755 every 8 connection(s)
I20260926 04:11:37.280958 19116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:37.281539 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19116
I20260926 04:11:37.281654 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:11:37.281874 855 external_mini_cluster.cc:1428] Setting key 5d5eb562196ba06368de79b93acb02b1
Sep 26 04:11:37 dist-test-slave-xf24 krb5kdc[18875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:11:37.294270 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.281731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:54573 (local address 127.0.213.254:40005)
0926 04:11:37.281803 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.281806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.282032 (+ 226us) server_negotiation.cc:407] Connection header received
0926 04:11:37.283545 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.283546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.283556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.283653 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:37.284655 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.285440 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.287187 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.287436 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.287501 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.290679 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:37.290681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:37.290681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:37.290689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:37.292425 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.293070 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.293071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.293071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.293130 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.293444 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.293446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.293446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.293628 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:37.293698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.294103 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.294192 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:11:37.294488 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.281698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40005 (local address 127.0.213.194:54573)
0926 04:11:37.281890 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.281946 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.283221 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.283763 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.283768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.284272 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:37.284558 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.284564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.285611 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.285613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.287035 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.287039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.287538 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.287540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.287704 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.288437 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:37.288534 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:37.290536 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:37.292682 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:37.292692 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:37.292694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:37.292988 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:37.293217 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:37.293218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:37.293218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:37.293343 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:37.293722 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:37.293729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:37.293915 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.294046 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.294206 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
I20260926 04:11:37.295162 19245 heartbeater.cc:347] Connected to a master server at 127.0.213.254:40005
I20260926 04:11:37.295298 19245 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:37.295550 19245 heartbeater.cc:511] Master 127.0.213.254:40005 requested a full tablet report, sending...
I20260926 04:11:37.296172 18920 ts_manager.cc:194] Registered new tserver with Master: bcf2863e70c14fab83669368754489df (127.0.213.194:40755)
I20260926 04:11:37.296756 18920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:54573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:11:37.312484 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40005
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:37.397573 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:37.397799 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:37.397825 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:37.399039 19250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:37.399083 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:37.399109 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:37.400674 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:40005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:37.400884 19250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:37.401144 19250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:37.406970 19258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.406987 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.406970 19256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:37.407217 19250 server_base.cc:1034] running on GCE node
I20260926 04:11:37.407444 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:37.407737 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:37.408914 19250 hybrid_clock.cc:648] HybridClock initialized: now 1790395897408907 us; error 44 us; skew 500 ppm
Sep 26 04:11:37 dist-test-slave-xf24 krb5kdc[18875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:37.411530 19250 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:11:37.412130 19250 webserver.cc:460] Webserver started at http://127.0.213.195:42017/ using document root <none> and password file <none>
I20260926 04:11:37.412375 19250 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:37.412441 19250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:37.412580 19250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:37.413791 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "0730ec4e351f4c63bb14f5d5408557cd"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "b4706fbe90ef8a12fe8a75b1c3293a3b"
server_key_iv: "c421ba5760976ccf254e2048a02bf2b5"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.414216 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "0730ec4e351f4c63bb14f5d5408557cd"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "b4706fbe90ef8a12fe8a75b1c3293a3b"
server_key_iv: "c421ba5760976ccf254e2048a02bf2b5"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.415854 19250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:37.416743 19265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.416926 19250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:37.417027 19250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "0730ec4e351f4c63bb14f5d5408557cd"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "b4706fbe90ef8a12fe8a75b1c3293a3b"
server_key_iv: "c421ba5760976ccf254e2048a02bf2b5"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.417110 19250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:37.441802 19250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:37.444327 19250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:37.444453 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:37.444636 19250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:37.444954 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:37.444988 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.445011 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:37.445057 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.450901 19250 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:43019
I20260926 04:11:37.450937 19378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:43019 every 8 connection(s)
I20260926 04:11:37.451268 19250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:11:37 dist-test-slave-xf24 krb5kdc[18875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:37.458333 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19250
I20260926 04:11:37.458455 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:11:37.458680 855 external_mini_cluster.cc:1428] Setting key 9e5a4594bac5a038d4a05f9be9031011
I20260926 04:11:37.461493 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.451990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:54821 (local address 127.0.213.254:40005)
0926 04:11:37.452040 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.452042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.452254 (+ 212us) server_negotiation.cc:407] Connection header received
0926 04:11:37.453175 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.453176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.453186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.453252 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:37.453937 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.454435 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.455540 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.455748 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.455817 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.458481 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:37.458483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:37.458483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:37.458492 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:37.460002 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.460481 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.460483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.460483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.460547 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.460815 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.460816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.460816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.460936 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:37.461013 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.461314 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.461402 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:11:37.461699 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.451933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:40005 (local address 127.0.213.195:54821)
0926 04:11:37.452120 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.452163 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.453071 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.453333 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.453337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.453651 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:37.453851 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.453856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.454525 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.454526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.455421 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.455424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.455848 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.455849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.455971 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.456641 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:37.456650 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:37.458354 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:37.460144 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:37.460153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:37.460155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:37.460388 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:37.460618 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:37.460619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:37.460619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:37.460739 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:37.461004 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:37.461011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:37.461169 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.461280 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.461453 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260926 04:11:37.462183 19379 heartbeater.cc:347] Connected to a master server at 127.0.213.254:40005
I20260926 04:11:37.462298 19379 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:37.462489 19379 heartbeater.cc:511] Master 127.0.213.254:40005 requested a full tablet report, sending...
I20260926 04:11:37.462846 18920 ts_manager.cc:194] Registered new tserver with Master: 0730ec4e351f4c63bb14f5d5408557cd (127.0.213.195:43019)
I20260926 04:11:37.463245 18920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:54821
I20260926 04:11:37.470484 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:11:37.478773 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.471952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.0.213.254:40005)
0926 04:11:37.472044 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.472047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.472182 (+ 135us) server_negotiation.cc:407] Connection header received
0926 04:11:37.472284 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.472286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.472296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.472371 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:37.473005 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.473506 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.474250 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.474454 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.474533 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.475909 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:37.475911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:37.475911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:37.475919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:37.477317 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.477768 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.477769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.477770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.477833 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:37.478089 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:37.478090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:37.478091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:37.478284 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:37.478425 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.478614 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.478675 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 04:11:37.480640 18920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:11:37.482213 18920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:11:37.489589 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.484750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:37401 (local address 127.0.0.1:50664)
0926 04:11:37.484924 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.484964 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.485033 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.485355 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.485359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.485361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:37.485450 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.485452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.486955 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.486957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.488413 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.488415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.489241 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.489242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.489328 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.489344 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.489415 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.489471 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":99,"thread_start_us":64,"threads_started":1}
I20260926 04:11:37.489619 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.484994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.0.213.193:37401)
0926 04:11:37.485195 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.485199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.485210 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:11:37.485260 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.485261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.485274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.485364 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:37.485666 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.486814 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.488505 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.489108 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.489239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.489311 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.489441 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.489525 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260926 04:11:37.490180 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.485020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.213.195:43019)
0926 04:11:37.485199 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.485203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.485451 (+ 248us) server_negotiation.cc:407] Connection header received
0926 04:11:37.485653 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.485656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.485673 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.485751 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:37.485837 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.486853 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.488491 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.489279 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.489342 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.489396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.489572 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.489623 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260926 04:11:37.490495 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.485031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.213.194:40755)
0926 04:11:37.485277 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.485281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.485296 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:11:37.485418 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.485422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.485438 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.485519 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:37.486969 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.487861 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.489183 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.490166 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.490238 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.490288 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.490381 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.490424 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
I20260926 04:11:37.490485 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.484750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:40755 (local address 127.0.0.1:60044)
0926 04:11:37.485248 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.485259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.485309 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.485518 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.485519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.485522 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:37.485578 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.485580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.487964 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.487966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.489089 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.489092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.490240 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.490241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.490313 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.490316 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.490371 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.490417 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260926 04:11:37.490375 19045 tablet_service.cc:1467] Processing CreateTablet for tablet dfe17c4a5a24420a860d5a9e062d9ba0 (DEFAULT_TABLE table=test-table [id=dec65386f34c4bf4ac75fa149df397eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:37.490793 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe17c4a5a24420a860d5a9e062d9ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:37.490876 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.484750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43019 (local address 127.0.0.1:50790)
0926 04:11:37.485396 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.485404 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.485477 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.485745 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.485746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.485748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:37.485788 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.485790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.486957 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.486958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.488395 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.488399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.489431 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.489432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.489483 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.489486 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.490432 (+ 946us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.490472 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":39,"spinlock_wait_cycles":896,"thread_start_us":97,"threads_started":1}
I20260926 04:11:37.491189 19179 tablet_service.cc:1467] Processing CreateTablet for tablet dfe17c4a5a24420a860d5a9e062d9ba0 (DEFAULT_TABLE table=test-table [id=dec65386f34c4bf4ac75fa149df397eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:37.491523 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe17c4a5a24420a860d5a9e062d9ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:37.491587 19313 tablet_service.cc:1467] Processing CreateTablet for tablet dfe17c4a5a24420a860d5a9e062d9ba0 (DEFAULT_TABLE table=test-table [id=dec65386f34c4bf4ac75fa149df397eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:37.491859 19313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe17c4a5a24420a860d5a9e062d9ba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:37.493742 19396 tablet_bootstrap.cc:492] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: Bootstrap starting.
I20260926 04:11:37.494405 19396 tablet_bootstrap.cc:654] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:37.494973 19396 log.cc:824] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:37.495128 19397 tablet_bootstrap.cc:492] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Bootstrap starting.
I20260926 04:11:37.495342 19398 tablet_bootstrap.cc:492] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Bootstrap starting.
I20260926 04:11:37.495806 19397 tablet_bootstrap.cc:654] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:37.495966 19398 tablet_bootstrap.cc:654] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:37.496590 19398 log.cc:824] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:37.496562 19397 log.cc:824] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:37.496958 19396 tablet_bootstrap.cc:492] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: No bootstrap required, opened a new log
I20260926 04:11:37.497043 19396 ts_tablet_manager.cc:1397] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:11:37.497328 19397 tablet_bootstrap.cc:492] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: No bootstrap required, opened a new log
I20260926 04:11:37.497349 19398 tablet_bootstrap.cc:492] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: No bootstrap required, opened a new log
I20260926 04:11:37.497398 19397 ts_tablet_manager.cc:1397] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:37.497424 19398 ts_tablet_manager.cc:1397] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:37.499007 19398 raft_consensus.cc:348] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.499185 19398 raft_consensus.cc:374] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:37.499229 19398 raft_consensus.cc:729] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0730ec4e351f4c63bb14f5d5408557cd, State: Initialized, Role: FOLLOWER
I20260926 04:11:37.499207 19396 raft_consensus.cc:348] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.499385 19396 raft_consensus.cc:374] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:37.499369 19398 consensus_queue.cc:260] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.499435 19396 raft_consensus.cc:729] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89033d2a5ab4993b34935ed938bdd63, State: Initialized, Role: FOLLOWER
I20260926 04:11:37.499424 19397 raft_consensus.cc:348] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.499548 19396 consensus_queue.cc:260] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.499578 19397 raft_consensus.cc:374] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:37.499634 19397 raft_consensus.cc:729] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf2863e70c14fab83669368754489df, State: Initialized, Role: FOLLOWER
I20260926 04:11:37.499691 19398 ts_tablet_manager.cc:1428] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:37.499743 19397 consensus_queue.cc:260] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.499814 19379 heartbeater.cc:503] Master 127.0.213.254:40005 was elected leader, sending a full tablet report...
I20260926 04:11:37.499830 19396 ts_tablet_manager.cc:1428] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:37.499957 19111 heartbeater.cc:503] Master 127.0.213.254:40005 was elected leader, sending a full tablet report...
I20260926 04:11:37.500005 19397 ts_tablet_manager.cc:1428] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:37.500185 19245 heartbeater.cc:503] Master 127.0.213.254:40005 was elected leader, sending a full tablet report...
W20260926 04:11:37.531522 19246 tablet.cc:2369] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:37.595527 19403 raft_consensus.cc:482] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:37.595662 19403 raft_consensus.cc:504] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.596187 19403 leader_election.cc:288] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0730ec4e351f4c63bb14f5d5408557cd (127.0.213.195:43019), bcf2863e70c14fab83669368754489df (127.0.213.194:40755)
I20260926 04:11:37.600554 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.596503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:44159 (local address 127.0.213.195:43019)
0926 04:11:37.596555 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.596558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.596630 (+ 72us) server_negotiation.cc:407] Connection header received
0926 04:11:37.596734 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.596735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.596745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.596802 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:37.597094 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.597840 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.599471 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.600139 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.600200 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.600210 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.600421 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.600475 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:11:37.600557 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.596409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:40755 (local address 127.0.213.193:51109)
0926 04:11:37.596505 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.596519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.596573 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.596891 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.596892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.596895 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:37.596982 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.596982 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.597973 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.597974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.599352 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.599355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.600249 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.600251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.600360 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.600365 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.600428 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.600479 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":86}
I20260926 04:11:37.600826 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.596429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:43019 (local address 127.0.213.193:44159)
0926 04:11:37.596570 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.596578 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.596654 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.596918 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.596918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.596919 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:37.596980 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.596982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.597973 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.597974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.599352 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.599355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.600232 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.600233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.600308 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.600314 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.600701 (+ 387us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.600751 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":31,"thread_start_us":79,"threads_started":1}
I20260926 04:11:37.601205 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.596482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:51109 (local address 127.0.213.194:40755)
0926 04:11:37.596558 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.596560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.596571 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:11:37.596658 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.596660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.596672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.596751 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:37.597094 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.597874 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.599471 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.600113 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.600200 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.600210 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.600473 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.600516 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:11:37.601289 19333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe17c4a5a24420a860d5a9e062d9ba0" candidate_uuid: "a89033d2a5ab4993b34935ed938bdd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0730ec4e351f4c63bb14f5d5408557cd" is_pre_election: true
I20260926 04:11:37.601565 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe17c4a5a24420a860d5a9e062d9ba0" candidate_uuid: "a89033d2a5ab4993b34935ed938bdd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf2863e70c14fab83669368754489df" is_pre_election: true
I20260926 04:11:37.601696 19199 raft_consensus.cc:2454] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89033d2a5ab4993b34935ed938bdd63 in term 0.
I20260926 04:11:37.601691 19333 raft_consensus.cc:2454] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89033d2a5ab4993b34935ed938bdd63 in term 0.
I20260926 04:11:37.601954 19001 leader_election.cc:302] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a89033d2a5ab4993b34935ed938bdd63, bcf2863e70c14fab83669368754489df; no voters:
I20260926 04:11:37.602166 19403 raft_consensus.cc:2781] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:11:37.602228 19403 raft_consensus.cc:482] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:11:37.602255 19403 raft_consensus.cc:3037] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:37.603176 19403 raft_consensus.cc:504] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.603410 19403 leader_election.cc:288] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [CANDIDATE]: Term 1 election: Requested vote from peers 0730ec4e351f4c63bb14f5d5408557cd (127.0.213.195:43019), bcf2863e70c14fab83669368754489df (127.0.213.194:40755)
I20260926 04:11:37.603483 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe17c4a5a24420a860d5a9e062d9ba0" candidate_uuid: "a89033d2a5ab4993b34935ed938bdd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf2863e70c14fab83669368754489df"
I20260926 04:11:37.603540 19333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe17c4a5a24420a860d5a9e062d9ba0" candidate_uuid: "a89033d2a5ab4993b34935ed938bdd63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0730ec4e351f4c63bb14f5d5408557cd"
I20260926 04:11:37.603561 19199 raft_consensus.cc:3037] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:37.603631 19333 raft_consensus.cc:3037] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:37.604532 19333 raft_consensus.cc:2454] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89033d2a5ab4993b34935ed938bdd63 in term 1.
I20260926 04:11:37.604532 19199 raft_consensus.cc:2454] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89033d2a5ab4993b34935ed938bdd63 in term 1.
I20260926 04:11:37.604709 19001 leader_election.cc:302] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a89033d2a5ab4993b34935ed938bdd63, bcf2863e70c14fab83669368754489df; no voters:
I20260926 04:11:37.604815 19403 raft_consensus.cc:2781] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:37.604992 19403 raft_consensus.cc:686] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [term 1 LEADER]: Becoming Leader. State: Replica: a89033d2a5ab4993b34935ed938bdd63, State: Running, Role: LEADER
I20260926 04:11:37.605103 19403 consensus_queue.cc:237] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } }
I20260926 04:11:37.606001 18918 catalog_manager.cc:5495] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 reported cstate change: term changed from 0 to 1, leader changed from <none> to a89033d2a5ab4993b34935ed938bdd63 (127.0.213.193). New cstate: current_term: 1 leader_uuid: "a89033d2a5ab4993b34935ed938bdd63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a89033d2a5ab4993b34935ed938bdd63" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37401 } health_report { overall_health: HEALTHY } } }
W20260926 04:11:37.617516 19112 tablet.cc:2369] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:37.635363 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.633141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.0.213.193:37401)
0926 04:11:37.633197 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.633199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.633208 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:37.633260 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.633261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.633279 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.633343 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:11:37.633441 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.633908 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.634661 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.634872 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.634938 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.635067 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:11:37.635154 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:11:37.635226 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.635276 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.635286 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 04:11:37.637274 19333 raft_consensus.cc:1261] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 1 FOLLOWER]: Refusing update from remote peer a89033d2a5ab4993b34935ed938bdd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:37.637290 19199 raft_consensus.cc:1261] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 1 FOLLOWER]: Refusing update from remote peer a89033d2a5ab4993b34935ed938bdd63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:37.637655 19403 consensus_queue.cc:1035] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf2863e70c14fab83669368754489df" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 40755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:37.637795 19406 consensus_queue.cc:1035] T dfe17c4a5a24420a860d5a9e062d9ba0 P a89033d2a5ab4993b34935ed938bdd63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0730ec4e351f4c63bb14f5d5408557cd" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 43019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:37.640338 19411 mvcc.cc:205] Tried to move back new op lower bound from 7333461596719636480 to 7333461596591140864. Current Snapshot: MvccSnapshot[applied={T|T < 7333461596719636480}]
I20260926 04:11:37.640682 19410 mvcc.cc:205] Tried to move back new op lower bound from 7333461596719636480 to 7333461596591140864. Current Snapshot: MvccSnapshot[applied={T|T < 7333461596719636480 or (T in {7333461596719636480})}]
I20260926 04:11:37.640098 19409 mvcc.cc:205] Tried to move back new op lower bound from 7333461596719636480 to 7333461596591140864. Current Snapshot: MvccSnapshot[applied={T|T < 7333461596719636480}]
I20260926 04:11:37.643260 18919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:37.643347 18919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:37.644273 18919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fda494e489844b46a77e5a64b98f6710: Sending DeleteTablet for 3 replicas of tablet dfe17c4a5a24420a860d5a9e062d9ba0
I20260926 04:11:37.644637 19045 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe17c4a5a24420a860d5a9e062d9ba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:37 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:50664
I20260926 04:11:37.644701 19313 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe17c4a5a24420a860d5a9e062d9ba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:37 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:50790
I20260926 04:11:37.644698 19179 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe17c4a5a24420a860d5a9e062d9ba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:37 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:60044
I20260926 04:11:37.644951 19421 tablet_replica.cc:330] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: stopping tablet replica
I20260926 04:11:37.645107 19421 raft_consensus.cc:2229] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:37.645215 19421 raft_consensus.cc:2258] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:37.645388 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18982
I20260926 04:11:37.646389 19421 ts_tablet_manager.cc:1905] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:37.647697 19420 tablet_replica.cc:330] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: stopping tablet replica
I20260926 04:11:37.647828 19420 raft_consensus.cc:2229] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:37.647943 19420 raft_consensus.cc:2258] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:37.647957 19421 ts_tablet_manager.cc:1918] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:37.648046 19421 log.cc:1196] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/dfe17c4a5a24420a860d5a9e062d9ba0
I20260926 04:11:37.648314 19421 ts_tablet_manager.cc:1939] T dfe17c4a5a24420a860d5a9e062d9ba0 P bcf2863e70c14fab83669368754489df: Deleting consensus metadata
I20260926 04:11:37.648758 18907 catalog_manager.cc:4841] TS bcf2863e70c14fab83669368754489df (127.0.213.194:40755): tablet dfe17c4a5a24420a860d5a9e062d9ba0 (table test-table [id=dec65386f34c4bf4ac75fa149df397eb]) successfully deleted
W20260926 04:11:37.651544 18906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:37401 (error 108)
I20260926 04:11:37.651794 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19116
I20260926 04:11:37.651881 19420 ts_tablet_manager.cc:1905] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:37.652181 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.652032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:37401 (local address 127.0.0.1:50684)
0926 04:11:37.652107 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.652140 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:37401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 04:11:37.653152 18906 catalog_manager.cc:4584] TS a89033d2a5ab4993b34935ed938bdd63 (127.0.213.193:37401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfe17c4a5a24420a860d5a9e062d9ba0: Network error: Client connection negotiation failed: client connection to 127.0.213.193:37401: connect: Connection refused (error 111)
I20260926 04:11:37.653533 19420 ts_tablet_manager.cc:1918] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:37.653630 19420 log.cc:1196] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/dfe17c4a5a24420a860d5a9e062d9ba0
I20260926 04:11:37.653875 19420 ts_tablet_manager.cc:1939] T dfe17c4a5a24420a860d5a9e062d9ba0 P 0730ec4e351f4c63bb14f5d5408557cd: Deleting consensus metadata
I20260926 04:11:37.654307 18905 catalog_manager.cc:4841] TS 0730ec4e351f4c63bb14f5d5408557cd (127.0.213.195:43019): tablet dfe17c4a5a24420a860d5a9e062d9ba0 (table test-table [id=dec65386f34c4bf4ac75fa149df397eb]) successfully deleted
I20260926 04:11:37.657639 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19250
I20260926 04:11:37.662197 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 18888
2026-09-26T04:11:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2080 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T04:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:37Z Disabled control of system clock
I20260926 04:11:37.690567 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:38973
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:37.772959 19426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:37.773198 19426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:37.773221 19426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:37.774394 19426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:37.774448 19426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:37.774461 19426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:37.774474 19426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:37.774487 19426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:37.775979 19426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:38973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:37.776243 19426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:37.776536 19426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:37.782034 19434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.782019 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.782019 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:37.782500 19426 server_base.cc:1034] running on GCE node
I20260926 04:11:37.782811 19426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:37.783143 19426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:37.784335 19426 hybrid_clock.cc:648] HybridClock initialized: now 1790395897784336 us; error 51 us; skew 500 ppm
I20260926 04:11:37.785830 19426 webserver.cc:460] Webserver started at http://127.0.213.254:46161/ using document root <none> and password file <none>
I20260926 04:11:37.786119 19426 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:37.786196 19426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:37.786365 19426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:37.787500 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "c8e9786b78d147fab306ec2217fb90a1"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "65b769243c3031643c98bd3fe2b22644"
server_key_iv: "b2e80d17d08b416e0c2d4e63a643e639"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.787878 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "c8e9786b78d147fab306ec2217fb90a1"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "65b769243c3031643c98bd3fe2b22644"
server_key_iv: "b2e80d17d08b416e0c2d4e63a643e639"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.789304 19426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:37.790170 19440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.790369 19426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:37.790486 19426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "c8e9786b78d147fab306ec2217fb90a1"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "65b769243c3031643c98bd3fe2b22644"
server_key_iv: "b2e80d17d08b416e0c2d4e63a643e639"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.790575 19426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:37.802161 19426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:37.802552 19426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:37.802772 19426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:37.807191 19426 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:38973
I20260926 04:11:37.807260 19492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:38973 every 8 connection(s)
I20260926 04:11:37.807638 19426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:37.808688 19493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:37.811014 19493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: Bootstrap starting.
I20260926 04:11:37.811545 19493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:37.811790 19493 log.cc:824] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:37.812532 19493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: No bootstrap required, opened a new log
I20260926 04:11:37.814085 19493 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e9786b78d147fab306ec2217fb90a1" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38973 } }
I20260926 04:11:37.814230 19493 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:37.814254 19493 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e9786b78d147fab306ec2217fb90a1, State: Initialized, Role: FOLLOWER
I20260926 04:11:37.814363 19493 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e9786b78d147fab306ec2217fb90a1" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38973 } }
I20260926 04:11:37.814446 19493 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:37.814483 19493 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:37.814532 19493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:37.815093 19493 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e9786b78d147fab306ec2217fb90a1" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38973 } }
I20260926 04:11:37.815215 19493 leader_election.cc:302] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e9786b78d147fab306ec2217fb90a1; no voters:
I20260926 04:11:37.815338 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19426
I20260926 04:11:37.815421 19493 leader_election.cc:288] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:37.815455 19496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:37.815513 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:37.815582 19496 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [term 1 LEADER]: Becoming Leader. State: Replica: c8e9786b78d147fab306ec2217fb90a1, State: Running, Role: LEADER
I20260926 04:11:37.815681 19496 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e9786b78d147fab306ec2217fb90a1" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38973 } }
I20260926 04:11:37.815716 19493 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:37.815749 855 external_mini_cluster.cc:1428] Setting key 4f9d430e161a1b4e16b29715c8980c6e
I20260926 04:11:37.817684 19498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e9786b78d147fab306ec2217fb90a1. Latest consensus state: current_term: 1 leader_uuid: "c8e9786b78d147fab306ec2217fb90a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e9786b78d147fab306ec2217fb90a1" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38973 } } }
I20260926 04:11:37.817790 19498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:37.818006 19497 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e9786b78d147fab306ec2217fb90a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e9786b78d147fab306ec2217fb90a1" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38973 } } }
I20260926 04:11:37.818076 19497 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1 [sys.catalog]: This master's current role is: LEADER
W20260926 04:11:37.818769 19512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:11:37.818828 19512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:11:37.818874 19513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:37.819012 19513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:37.820439 19513 catalog_manager.cc:1347] Generated new cluster ID: 23e7f621f45d4f319083a14366bcb45e
I20260926 04:11:37.820495 19513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:37.822381 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.816483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.213.254:38973)
0926 04:11:37.816745 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.816755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.816800 (+ 45us) server_negotiation.cc:407] Connection header received
0926 04:11:37.816942 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.816950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.817023 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.817192 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:37.817639 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.819081 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.820207 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.820795 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.820907 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.821680 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:37.821684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:37.821684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:37.821883 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:37.821922 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:37.821936 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:37.821998 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.822092 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.822166 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260926 04:11:37.833599 19513 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:37.834177 19513 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:37.844031 19513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: Generated new TSK 0
I20260926 04:11:37.844270 19513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:37.848371 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:38973
--builtin_ntp_servers=127.0.213.212:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:37.930240 19517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:37.930459 19517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:37.930481 19517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:37.931681 19517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:37.931723 19517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:37.931748 19517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:37.933260 19517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:37.933537 19517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:37.933873 19517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:37.938985 19523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:37.939015 19522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:37.939253 19517 server_base.cc:1034] running on GCE node
W20260926 04:11:37.939021 19525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:37.939620 19517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:37.939893 19517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:37.941062 19517 hybrid_clock.cc:648] HybridClock initialized: now 1790395897941037 us; error 40 us; skew 500 ppm
I20260926 04:11:37.942514 19517 webserver.cc:460] Webserver started at http://127.0.213.193:35693/ using document root <none> and password file <none>
I20260926 04:11:37.942794 19517 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:37.942843 19517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:37.942960 19517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:37.944022 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "e5075c8d6b904977a9f0d835ae02b6ca"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "566b033152d163887b8e00f1eedd27b2"
server_key_iv: "66a021ed9e2554ba0d1290925076ed88"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.944449 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "e5075c8d6b904977a9f0d835ae02b6ca"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "566b033152d163887b8e00f1eedd27b2"
server_key_iv: "66a021ed9e2554ba0d1290925076ed88"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.945721 19517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:37.946527 19531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.946833 19517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:37.946946 19517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "e5075c8d6b904977a9f0d835ae02b6ca"
format_stamp: "Formatted at 2026-09-26 04:11:37 on dist-test-slave-xf24"
server_key: "566b033152d163887b8e00f1eedd27b2"
server_key_iv: "66a021ed9e2554ba0d1290925076ed88"
server_key_version: "encryptionkey@0"
I20260926 04:11:37.947038 19517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:37.956861 19517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:37.957202 19517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:37.957353 19517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:37.957587 19517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:37.957921 19517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:37.957984 19517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.958025 19517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:37.958079 19517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:37.963987 19517 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:36389
I20260926 04:11:37.964094 19644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:36389 every 8 connection(s)
I20260926 04:11:37.964460 19517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:11:37.969859 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.965232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:52131 (local address 127.0.213.254:38973)
0926 04:11:37.965309 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:37.965311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:37.965545 (+ 234us) server_negotiation.cc:407] Connection header received
0926 04:11:37.966052 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:37.966053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:37.966064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:37.966134 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:37.966890 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.967414 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.968558 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.968767 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.968828 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.969623 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:37.969624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:37.969625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:37.969629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:37.969642 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:37.969646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:37.969689 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:37.969771 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 04:11:37.969799 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:11:37.970049 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:37.965230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:38973 (local address 127.0.213.193:52131)
0926 04:11:37.965412 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:37.965454 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:37.965963 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:37.966210 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:37.966213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:37.966222 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:37.966774 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.966778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.967510 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.967511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.968455 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:37.968458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:37.968848 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:37.968848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:37.968952 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:37.969448 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:37.969448 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:37.969476 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:37.969498 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:37.969703 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:37.969708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:37.969710 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:11:37.969763 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:11:37.969843 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260926 04:11:37.970604 19645 heartbeater.cc:347] Connected to a master server at 127.0.213.254:38973
I20260926 04:11:37.970710 19645 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:37.970887 19645 heartbeater.cc:511] Master 127.0.213.254:38973 requested a full tablet report, sending...
I20260926 04:11:37.971395 19457 ts_manager.cc:194] Registered new tserver with Master: e5075c8d6b904977a9f0d835ae02b6ca (127.0.213.193:36389)
I20260926 04:11:37.971971 19457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.193:52131
I20260926 04:11:37.973320 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19517
I20260926 04:11:37.973400 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:11:37.973549 855 external_mini_cluster.cc:1428] Setting key 7c41291b78fb49a251a42adbc4f70d98
I20260926 04:11:37.974436 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:38973
--builtin_ntp_servers=127.0.213.212:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:38.057456 19648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:38.057694 19648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:38.057718 19648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:38.058892 19648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:38.058928 19648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:38.059010 19648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:38.060431 19648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:38.060636 19648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:38.060892 19648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:38.065994 19656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:38.066012 19653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:38.066005 19654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:38.066246 19648 server_base.cc:1034] running on GCE node
I20260926 04:11:38.066496 19648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:38.066761 19648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:38.067904 19648 hybrid_clock.cc:648] HybridClock initialized: now 1790395898067884 us; error 43 us; skew 500 ppm
I20260926 04:11:38.069355 19648 webserver.cc:460] Webserver started at http://127.0.213.194:45689/ using document root <none> and password file <none>
I20260926 04:11:38.069636 19648 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:38.069687 19648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:38.069828 19648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:38.070925 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "0dfff0d5b084408190113a25fa3b6b87"
format_stamp: "Formatted at 2026-09-26 04:11:38 on dist-test-slave-xf24"
server_key: "16f2d334f3e4b3623db5d2c23bdc7fca"
server_key_iv: "6ec83689a6233fc7c8021f9fbecd869e"
server_key_version: "encryptionkey@0"
I20260926 04:11:38.071292 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "0dfff0d5b084408190113a25fa3b6b87"
format_stamp: "Formatted at 2026-09-26 04:11:38 on dist-test-slave-xf24"
server_key: "16f2d334f3e4b3623db5d2c23bdc7fca"
server_key_iv: "6ec83689a6233fc7c8021f9fbecd869e"
server_key_version: "encryptionkey@0"
I20260926 04:11:38.072719 19648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:11:38.073442 19662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.073648 19648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.073740 19648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "0dfff0d5b084408190113a25fa3b6b87"
format_stamp: "Formatted at 2026-09-26 04:11:38 on dist-test-slave-xf24"
server_key: "16f2d334f3e4b3623db5d2c23bdc7fca"
server_key_iv: "6ec83689a6233fc7c8021f9fbecd869e"
server_key_version: "encryptionkey@0"
I20260926 04:11:38.073814 19648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:38.089851 19648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:38.090194 19648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:38.090334 19648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:38.090605 19648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:38.090955 19648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:38.091010 19648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.091053 19648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:38.091080 19648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.097421 19648 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:44345
I20260926 04:11:38.097518 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:44345 every 8 connection(s)
I20260926 04:11:38.097899 19648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:38.099776 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19648
I20260926 04:11:38.099882 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:11:38.100132 855 external_mini_cluster.cc:1428] Setting key 3cd8f91ed9ce9948179ff8e811f655e0
I20260926 04:11:38.101286 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:38973
--builtin_ntp_servers=127.0.213.212:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:11:38.103973 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.098734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:60819 (local address 127.0.213.254:38973)
0926 04:11:38.098784 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.098790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.099030 (+ 240us) server_negotiation.cc:407] Connection header received
0926 04:11:38.099579 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.099581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.099592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.099664 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:38.100664 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.101266 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.102405 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.102613 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.102697 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.103674 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:38.103676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:38.103676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:38.103681 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:38.103704 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:38.103708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:38.103768 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.103849 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.103883 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:11:38.104231 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.098699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:38973 (local address 127.0.213.194:60819)
0926 04:11:38.098889 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.098935 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.099423 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.099773 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.099778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.099790 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:38.100534 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.100541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.101373 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.101374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.102281 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.102283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.102711 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.102712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.102875 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.103508 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:38.103509 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:38.103540 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:38.103567 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:38.103771 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:38.103777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:38.103780 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.103854 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.103967 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 04:11:38.104679 19776 heartbeater.cc:347] Connected to a master server at 127.0.213.254:38973
I20260926 04:11:38.104779 19776 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:38.104988 19776 heartbeater.cc:511] Master 127.0.213.254:38973 requested a full tablet report, sending...
I20260926 04:11:38.105373 19457 ts_manager.cc:194] Registered new tserver with Master: 0dfff0d5b084408190113a25fa3b6b87 (127.0.213.194:44345)
I20260926 04:11:38.105767 19457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.194:60819
W20260926 04:11:38.186458 19779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:38.186714 19779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:38.186753 19779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:38.187922 19779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:38.187983 19779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:38.188028 19779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:38.189424 19779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:38.189698 19779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:38.190006 19779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:38.195729 19787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:38.195744 19784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:38.195737 19785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:38.196638 19779 server_base.cc:1034] running on GCE node
I20260926 04:11:38.196928 19779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:38.197253 19779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:38.198432 19779 hybrid_clock.cc:648] HybridClock initialized: now 1790395898198374 us; error 72 us; skew 500 ppm
I20260926 04:11:38.199878 19779 webserver.cc:460] Webserver started at http://127.0.213.195:38437/ using document root <none> and password file <none>
I20260926 04:11:38.200244 19779 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:38.200325 19779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:38.200485 19779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:38.201658 19779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "3f148392d67443fb8aff601fac06e138"
format_stamp: "Formatted at 2026-09-26 04:11:38 on dist-test-slave-xf24"
server_key: "4ee74bb0e79c125e60ca2582498cef4c"
server_key_iv: "e206e614386ed11cb3380e66f2bf3d07"
server_key_version: "encryptionkey@0"
I20260926 04:11:38.202044 19779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "3f148392d67443fb8aff601fac06e138"
format_stamp: "Formatted at 2026-09-26 04:11:38 on dist-test-slave-xf24"
server_key: "4ee74bb0e79c125e60ca2582498cef4c"
server_key_iv: "e206e614386ed11cb3380e66f2bf3d07"
server_key_version: "encryptionkey@0"
I20260926 04:11:38.203495 19779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:38.204368 19793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.204607 19779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:38.204718 19779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "3f148392d67443fb8aff601fac06e138"
format_stamp: "Formatted at 2026-09-26 04:11:38 on dist-test-slave-xf24"
server_key: "4ee74bb0e79c125e60ca2582498cef4c"
server_key_iv: "e206e614386ed11cb3380e66f2bf3d07"
server_key_version: "encryptionkey@0"
I20260926 04:11:38.204794 19779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:38.213346 19779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:38.213704 19779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:38.213862 19779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:38.214095 19779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:38.214452 19779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:38.214510 19779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.214561 19779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:38.214587 19779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:38.220944 19779 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:40979
I20260926 04:11:38.220983 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:40979 every 8 connection(s)
I20260926 04:11:38.221346 19779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:11:38.226776 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19779
I20260926 04:11:38.226904 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:11:38.226992 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.222227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:45329 (local address 127.0.213.254:38973)
0926 04:11:38.222321 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.222324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.222504 (+ 180us) server_negotiation.cc:407] Connection header received
0926 04:11:38.223006 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.223007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.223020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.223080 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:38.223757 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.224350 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.225340 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.225523 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.225581 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.226531 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:38.226533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:38.226533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:38.226537 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:38.226554 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:38.226558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:38.226611 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.226754 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.226781 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:11:38.227128 855 external_mini_cluster.cc:1428] Setting key 64cd619acdb638744ae00fa863a6c566
I20260926 04:11:38.227164 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.222154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:38973 (local address 127.0.213.195:45329)
0926 04:11:38.222362 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.222397 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.222917 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.223139 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.223143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.223151 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:38.223656 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.223660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.224439 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.224440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.225256 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.225259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.225623 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.225624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.225746 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.226340 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:38.226341 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:38.226372 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:38.226397 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:38.226636 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:38.226643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:38.226646 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.226740 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.226869 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260926 04:11:38.227733 19907 heartbeater.cc:347] Connected to a master server at 127.0.213.254:38973
I20260926 04:11:38.227856 19907 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:38.228124 19907 heartbeater.cc:511] Master 127.0.213.254:38973 requested a full tablet report, sending...
I20260926 04:11:38.228556 19457 ts_manager.cc:194] Registered new tserver with Master: 3f148392d67443fb8aff601fac06e138 (127.0.213.195:40979)
I20260926 04:11:38.228929 19457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.195:45329
I20260926 04:11:38.239508 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:11:38.244266 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.241166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.0.213.254:38973)
0926 04:11:38.241235 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.241236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.241332 (+ 96us) server_negotiation.cc:407] Connection header received
0926 04:11:38.241366 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.241367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.241376 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.241436 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:38.241581 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.242068 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.242854 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.243076 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.243162 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.243926 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:38.243928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:38.243928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:38.243933 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:38.243947 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:38.243952 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:38.244119 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.244180 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.244192 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:11:38.246181 19457 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:11:38.247848 19457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:11:38.254331 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.250542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.0.213.193:36389)
0926 04:11:38.250778 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.250783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.250793 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:11:38.250835 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.250837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.250851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.250958 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:38.251114 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.251983 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.253183 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.253943 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.254004 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.254107 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.254196 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.254234 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260926 04:11:38.254319 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.250269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36389 (local address 127.0.0.1:58430)
0926 04:11:38.250606 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.250618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.250670 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.250957 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.250958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.250960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:38.251013 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.251016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.252093 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.252094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.253077 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.253079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.254003 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.254004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.254081 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.254094 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.254181 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.254237 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":97,"threads_started":1}
I20260926 04:11:38.254581 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.250233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:44345 (local address 127.0.0.1:37036)
0926 04:11:38.250462 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.250509 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.250592 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.250930 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.250933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.250935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:38.251011 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.251014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.252405 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.252406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.253446 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.253449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.254337 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.254339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.254402 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.254406 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.254453 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.254495 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260926 04:11:38.254829 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.250309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.213.194:44345)
0926 04:11:38.250765 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.250769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.250778 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:38.250811 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.250813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.250825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.250915 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:38.251108 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.252308 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.253548 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.254146 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.254218 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.254302 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.254697 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.254745 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":118,"threads_started":1}
I20260926 04:11:38.255343 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.250478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:40979 (local address 127.0.0.1:57202)
0926 04:11:38.251078 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.251090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.251138 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.251399 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.251400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.251402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:38.251439 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.251441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.252813 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.252814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.253868 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.253871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.255071 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.255072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.255144 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.255150 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.255217 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.255264 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":71,"threads_started":1}
I20260926 04:11:38.255369 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.250575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.213.195:40979)
0926 04:11:38.250790 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.250794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.251199 (+ 405us) server_negotiation.cc:407] Connection header received
0926 04:11:38.251244 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.251247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.251262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.251331 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:38.251570 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.252714 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.253981 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.254996 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.255065 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.255126 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.255229 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.255259 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260926 04:11:38.255333 19579 tablet_service.cc:1467] Processing CreateTablet for tablet 3a356911e71841b4a30f7d60263de234 (DEFAULT_TABLE table=test-table [id=5ccc38cebb9f48458c1640ebba733919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:38.255774 19579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a356911e71841b4a30f7d60263de234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:38.256367 19841 tablet_service.cc:1467] Processing CreateTablet for tablet 3a356911e71841b4a30f7d60263de234 (DEFAULT_TABLE table=test-table [id=5ccc38cebb9f48458c1640ebba733919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:38.256695 19841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a356911e71841b4a30f7d60263de234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:38.258783 19921 tablet_bootstrap.cc:492] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Bootstrap starting.
I20260926 04:11:38.259493 19921 tablet_bootstrap.cc:654] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:38.259475 19710 tablet_service.cc:1467] Processing CreateTablet for tablet 3a356911e71841b4a30f7d60263de234 (DEFAULT_TABLE table=test-table [id=5ccc38cebb9f48458c1640ebba733919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:38.259824 19710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a356911e71841b4a30f7d60263de234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:38.259835 19921 log.cc:824] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:38.260461 19923 tablet_bootstrap.cc:492] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Bootstrap starting.
I20260926 04:11:38.260663 19921 tablet_bootstrap.cc:492] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: No bootstrap required, opened a new log
I20260926 04:11:38.260756 19921 ts_tablet_manager.cc:1397] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:38.261137 19923 tablet_bootstrap.cc:654] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:38.261473 19923 log.cc:824] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:38.262279 19923 tablet_bootstrap.cc:492] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: No bootstrap required, opened a new log
I20260926 04:11:38.262354 19923 ts_tablet_manager.cc:1397] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:38.263245 19921 raft_consensus.cc:348] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.263405 19921 raft_consensus.cc:374] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:38.263448 19921 raft_consensus.cc:729] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5075c8d6b904977a9f0d835ae02b6ca, State: Initialized, Role: FOLLOWER
I20260926 04:11:38.263566 19921 consensus_queue.cc:260] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.263845 19921 ts_tablet_manager.cc:1428] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:38.263981 19645 heartbeater.cc:503] Master 127.0.213.254:38973 was elected leader, sending a full tablet report...
I20260926 04:11:38.264985 19923 raft_consensus.cc:348] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.265131 19923 raft_consensus.cc:374] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:38.265179 19923 raft_consensus.cc:729] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f148392d67443fb8aff601fac06e138, State: Initialized, Role: FOLLOWER
I20260926 04:11:38.265288 19923 consensus_queue.cc:260] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.265532 19923 ts_tablet_manager.cc:1428] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:38.265681 19907 heartbeater.cc:503] Master 127.0.213.254:38973 was elected leader, sending a full tablet report...
I20260926 04:11:38.265793 19925 tablet_bootstrap.cc:492] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Bootstrap starting.
I20260926 04:11:38.266472 19925 tablet_bootstrap.cc:654] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:38.266816 19925 log.cc:824] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:38.267627 19925 tablet_bootstrap.cc:492] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: No bootstrap required, opened a new log
I20260926 04:11:38.267684 19925 ts_tablet_manager.cc:1397] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:38.269451 19925 raft_consensus.cc:348] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.269577 19925 raft_consensus.cc:374] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:38.269604 19925 raft_consensus.cc:729] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfff0d5b084408190113a25fa3b6b87, State: Initialized, Role: FOLLOWER
I20260926 04:11:38.269726 19925 consensus_queue.cc:260] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.269975 19925 ts_tablet_manager.cc:1428] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:11:38.270054 19776 heartbeater.cc:503] Master 127.0.213.254:38973 was elected leader, sending a full tablet report...
W20260926 04:11:38.348523 19777 tablet.cc:2369] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:38.425339 19926 raft_consensus.cc:482] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:38.425477 19926 raft_consensus.cc:504] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.425791 19926 leader_election.cc:288] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfff0d5b084408190113a25fa3b6b87 (127.0.213.194:44345), 3f148392d67443fb8aff601fac06e138 (127.0.213.195:40979)
I20260926 04:11:38.429035 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.426149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:39261 (local address 127.0.213.194:44345)
0926 04:11:38.426240 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.426242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.426253 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:11:38.426289 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.426289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.426300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.426387 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:38.426593 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.427159 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.428226 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.428704 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.428760 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.428766 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.428944 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.428976 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 04:11:38.429121 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.426056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:44345 (local address 127.0.213.193:39261)
0926 04:11:38.426117 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.426130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.426179 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.426406 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.426407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.426415 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:38.426512 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.426514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.427250 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.427252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.428143 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.428146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.428787 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.428788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.428864 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.428868 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.428940 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.429016 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 04:11:38.429404 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.426292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:55663 (local address 127.0.213.195:40979)
0926 04:11:38.426337 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.426339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.426392 (+ 53us) server_negotiation.cc:407] Connection header received
0926 04:11:38.426487 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.426488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.426498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.426569 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:38.426797 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.427371 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.428517 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.429012 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.429081 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.429089 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.429295 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.429339 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:11:38.429628 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.426179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:40979 (local address 127.0.213.193:55663)
0926 04:11:38.426339 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.426349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:38.426403 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:38.426591 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:38.426592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:38.426594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:38.426692 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.426693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.427503 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.427504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.428398 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:38.428401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.429102 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.429103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:38.429199 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.429203 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:38.429491 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 04:11:38.429542 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260926 04:11:38.429569 19730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a356911e71841b4a30f7d60263de234" candidate_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfff0d5b084408190113a25fa3b6b87" is_pre_election: true
I20260926 04:11:38.429764 19730 raft_consensus.cc:2454] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5075c8d6b904977a9f0d835ae02b6ca in term 0.
I20260926 04:11:38.429983 19533 leader_election.cc:302] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfff0d5b084408190113a25fa3b6b87, e5075c8d6b904977a9f0d835ae02b6ca; no voters:
I20260926 04:11:38.429958 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a356911e71841b4a30f7d60263de234" candidate_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f148392d67443fb8aff601fac06e138" is_pre_election: true
I20260926 04:11:38.430102 19861 raft_consensus.cc:2454] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5075c8d6b904977a9f0d835ae02b6ca in term 0.
I20260926 04:11:38.430133 19926 raft_consensus.cc:2781] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:11:38.430215 19926 raft_consensus.cc:482] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:11:38.430246 19926 raft_consensus.cc:3037] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:38.430955 19926 raft_consensus.cc:504] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.431094 19926 leader_election.cc:288] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [CANDIDATE]: Term 1 election: Requested vote from peers 0dfff0d5b084408190113a25fa3b6b87 (127.0.213.194:44345), 3f148392d67443fb8aff601fac06e138 (127.0.213.195:40979)
I20260926 04:11:38.431246 19730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a356911e71841b4a30f7d60263de234" candidate_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfff0d5b084408190113a25fa3b6b87"
I20260926 04:11:38.431281 19861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a356911e71841b4a30f7d60263de234" candidate_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f148392d67443fb8aff601fac06e138"
I20260926 04:11:38.431320 19730 raft_consensus.cc:3037] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:38.431366 19861 raft_consensus.cc:3037] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:38.432024 19730 raft_consensus.cc:2454] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5075c8d6b904977a9f0d835ae02b6ca in term 1.
I20260926 04:11:38.432147 19861 raft_consensus.cc:2454] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5075c8d6b904977a9f0d835ae02b6ca in term 1.
I20260926 04:11:38.432210 19533 leader_election.cc:302] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfff0d5b084408190113a25fa3b6b87, e5075c8d6b904977a9f0d835ae02b6ca; no voters:
I20260926 04:11:38.432313 19926 raft_consensus.cc:2781] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:38.432466 19926 raft_consensus.cc:686] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 1 LEADER]: Becoming Leader. State: Replica: e5075c8d6b904977a9f0d835ae02b6ca, State: Running, Role: LEADER
I20260926 04:11:38.432588 19926 consensus_queue.cc:237] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } }
I20260926 04:11:38.433540 19457 catalog_manager.cc:5495] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e5075c8d6b904977a9f0d835ae02b6ca (127.0.213.193). New cstate: current_term: 1 leader_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5075c8d6b904977a9f0d835ae02b6ca" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 } health_report { overall_health: UNKNOWN } } }
W20260926 04:11:38.465224 19646 tablet.cc:2369] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:11:38.471966 19908 tablet.cc:2369] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:38.483775 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.481234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.0.213.193:36389)
0926 04:11:38.481287 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:38.481289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:38.481297 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:11:38.481343 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:38.481344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:38.481364 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:38.481432 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:11:38.481635 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.482131 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.482931 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:38.483141 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:38.483221 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:38.483372 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:11:38.483475 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:11:38.483596 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:38.483649 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 04:11:38.483660 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 04:11:38.485754 19861 raft_consensus.cc:1261] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 1 FOLLOWER]: Refusing update from remote peer e5075c8d6b904977a9f0d835ae02b6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:38.485754 19730 raft_consensus.cc:1261] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 1 FOLLOWER]: Refusing update from remote peer e5075c8d6b904977a9f0d835ae02b6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:38.486147 19926 consensus_queue.cc:1035] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfff0d5b084408190113a25fa3b6b87" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:38.486332 19931 consensus_queue.cc:1035] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f148392d67443fb8aff601fac06e138" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:38.490223 19934 mvcc.cc:205] Tried to move back new op lower bound from 7333461600194564096 to 7333461599980425216. Current Snapshot: MvccSnapshot[applied={T|T < 7333461600194564096}]
I20260926 04:11:38.492887 19456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:38.493003 19456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:38.494757 19456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8e9786b78d147fab306ec2217fb90a1: Sending DeleteTablet for 3 replicas of tablet 3a356911e71841b4a30f7d60263de234
I20260926 04:11:38.495144 19579 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a356911e71841b4a30f7d60263de234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:38 UTC) from {username='slave'} at 127.0.0.1:58430
I20260926 04:11:38.495411 19942 tablet_replica.cc:330] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: stopping tablet replica
I20260926 04:11:38.495545 19942 raft_consensus.cc:2229] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:11:38.495716 19942 raft_consensus.cc:2258] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:38.495666 19841 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a356911e71841b4a30f7d60263de234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:38 UTC) from {username='slave'} at 127.0.0.1:57202
I20260926 04:11:38.495689 19710 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a356911e71841b4a30f7d60263de234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:38 UTC) from {username='slave'} at 127.0.0.1:37036
I20260926 04:11:38.496325 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19517
I20260926 04:11:38.496323 19942 ts_tablet_manager.cc:1905] T 3a356911e71841b4a30f7d60263de234 P e5075c8d6b904977a9f0d835ae02b6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:38.497480 19944 tablet_replica.cc:330] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: stopping tablet replica
I20260926 04:11:38.497635 19944 raft_consensus.cc:2229] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:38.497714 19944 pending_rounds.cc:62] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Trying to abort 2 pending ops.
I20260926 04:11:38.497753 19944 pending_rounds.cc:68] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333461600194564096 op_type: WRITE_OP write_request { tablet_id: "3a356911e71841b4a30f7d60263de234" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\246\216\335\325\006\"3\n\005slave\022*\n 5ccc38cebb9f48458c1640ebba733919\020\001\030\001 \001(\001" signature: "<redacted>""\207\317\235\001*\315\217C\337e^\251\0173%\323h\356\271\275\242\336\026\213\216\374U\000\201\222\021\370\213\270\3640+\357\336\212\214E\341\202,r*\307i\250V/\242;\341\024=\320\373\013\303d\357\325" signing_key_seq_num: 0 } } request_id { client_id: "1b923a9210c74daca8734caad2c0adff" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 04:11:38.498023 19944 pending_rounds.cc:68] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333461599980425216 op_type: NO_OP noop_request { }
I20260926 04:11:38.498092 19944 raft_consensus.cc:2866] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 04:11:38.498121 19944 raft_consensus.cc:2258] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:38.498286 19943 tablet_replica.cc:330] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: stopping tablet replica
I20260926 04:11:38.498404 19943 raft_consensus.cc:2229] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:38.498570 19943 raft_consensus.cc:2258] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:38.499056 19943 ts_tablet_manager.cc:1905] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:38.500454 19944 ts_tablet_manager.cc:1905] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:38.500967 19943 ts_tablet_manager.cc:1918] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:38.501048 19943 log.cc:1196] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/3a356911e71841b4a30f7d60263de234
I20260926 04:11:38.501300 19943 ts_tablet_manager.cc:1939] T 3a356911e71841b4a30f7d60263de234 P 0dfff0d5b084408190113a25fa3b6b87: Deleting consensus metadata
I20260926 04:11:38.502027 19944 ts_tablet_manager.cc:1918] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:38.502116 19944 log.cc:1196] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/3a356911e71841b4a30f7d60263de234
I20260926 04:11:38.502148 19442 catalog_manager.cc:4841] TS 0dfff0d5b084408190113a25fa3b6b87 (127.0.213.194:44345): tablet 3a356911e71841b4a30f7d60263de234 (table test-table [id=5ccc38cebb9f48458c1640ebba733919]) successfully deleted
I20260926 04:11:38.502377 19944 ts_tablet_manager.cc:1939] T 3a356911e71841b4a30f7d60263de234 P 3f148392d67443fb8aff601fac06e138: Deleting consensus metadata
I20260926 04:11:38.502846 19442 catalog_manager.cc:4841] TS 3f148392d67443fb8aff601fac06e138 (127.0.213.195:40979): tablet 3a356911e71841b4a30f7d60263de234 (table test-table [id=5ccc38cebb9f48458c1640ebba733919]) successfully deleted
W20260926 04:11:38.504185 19441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:36389 (error 108)
I20260926 04:11:38.504393 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19648
I20260926 04:11:38.505439 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:38.504635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:36389 (local address 127.0.0.1:58446)
0926 04:11:38.505363 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:38.505408 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:36389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705}
W20260926 04:11:38.505582 19441 catalog_manager.cc:4584] TS e5075c8d6b904977a9f0d835ae02b6ca (127.0.213.193:36389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a356911e71841b4a30f7d60263de234: Network error: Client connection negotiation failed: client connection to 127.0.213.193:36389: connect: Connection refused (error 111)
I20260926 04:11:38.510632 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19779
I20260926 04:11:38.515071 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19426
2026-09-26T04:11:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (851 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:38 dist-test-slave-xf24 krb5kdc[19948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:38 dist-test-slave-xf24 krb5kdc[19948](info): set up 2 sockets
Sep 26 04:11:38 dist-test-slave-xf24 krb5kdc[19948](info): commencing operation
krb5kdc: starting...
W20260926 04:11:39.556135 855 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:39 dist-test-slave-xf24 krb5kdc[19948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:11:39.684674 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41391
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:39.767827 19961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:39.768121 19961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:39.768172 19961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:39.769395 19961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:39.769470 19961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:39.769495 19961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:39.769524 19961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:39.769556 19961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:39.771575 19961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:39.771890 19961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:39.772161 19961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:39.777442 19969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:39.777446 19967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:39.777442 19966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:39.777704 19961 server_base.cc:1034] running on GCE node
I20260926 04:11:39.778012 19961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:39.778363 19961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:39.779520 19961 hybrid_clock.cc:648] HybridClock initialized: now 1790395899779497 us; error 50 us; skew 500 ppm
Sep 26 04:11:39 dist-test-slave-xf24 krb5kdc[19948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:39.782330 19961 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:11:39.782845 19961 webserver.cc:460] Webserver started at http://127.0.213.254:36923/ using document root <none> and password file <none>
I20260926 04:11:39.783129 19961 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:39.783206 19961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:39.783361 19961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:39.784559 19961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "31eb10d64dd548b688a7adba2f8068cd"
format_stamp: "Formatted at 2026-09-26 04:11:39 on dist-test-slave-xf24"
server_key: "f2887c8299e3785b4f4d309e48144c22"
server_key_iv: "962ffce1871f1dab0b1159dfb23269fa"
server_key_version: "encryptionkey@0"
I20260926 04:11:39.784924 19961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "31eb10d64dd548b688a7adba2f8068cd"
format_stamp: "Formatted at 2026-09-26 04:11:39 on dist-test-slave-xf24"
server_key: "f2887c8299e3785b4f4d309e48144c22"
server_key_iv: "962ffce1871f1dab0b1159dfb23269fa"
server_key_version: "encryptionkey@0"
I20260926 04:11:39.786360 19961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:11:39.787165 19976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:39.787360 19961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:39.787462 19961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "31eb10d64dd548b688a7adba2f8068cd"
format_stamp: "Formatted at 2026-09-26 04:11:39 on dist-test-slave-xf24"
server_key: "f2887c8299e3785b4f4d309e48144c22"
server_key_iv: "962ffce1871f1dab0b1159dfb23269fa"
server_key_version: "encryptionkey@0"
I20260926 04:11:39.787545 19961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:39.805660 19961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:39.807956 19961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:39.808159 19961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:39.811985 19961 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:41391
I20260926 04:11:39.812043 20028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:41391 every 8 connection(s)
I20260926 04:11:39.812439 19961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:39.813395 20029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:39.815826 20029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd: Bootstrap starting.
I20260926 04:11:39.816510 20029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:39.816828 20029 log.cc:824] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:39.817633 20029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd: No bootstrap required, opened a new log
I20260926 04:11:39.819268 20029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb10d64dd548b688a7adba2f8068cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41391 } }
I20260926 04:11:39.819411 20029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:39.819433 20029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31eb10d64dd548b688a7adba2f8068cd, State: Initialized, Role: FOLLOWER
I20260926 04:11:39.819612 20029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb10d64dd548b688a7adba2f8068cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41391 } }
I20260926 04:11:39.819685 20029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:39.819708 20029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:39.819744 20029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:39.820147 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 19961
I20260926 04:11:39.820243 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:39.820456 855 external_mini_cluster.cc:1428] Setting key d8a256a8b3c9527165671ab4623e6608
I20260926 04:11:39.820554 20029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb10d64dd548b688a7adba2f8068cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41391 } }
I20260926 04:11:39.820688 20029 leader_election.cc:302] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31eb10d64dd548b688a7adba2f8068cd; no voters:
I20260926 04:11:39.820904 20029 leader_election.cc:288] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:39.820967 20032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:39.821134 20032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [term 1 LEADER]: Becoming Leader. State: Replica: 31eb10d64dd548b688a7adba2f8068cd, State: Running, Role: LEADER
I20260926 04:11:39.821169 20029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:39.821237 20032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb10d64dd548b688a7adba2f8068cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41391 } }
I20260926 04:11:39.821560 20036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31eb10d64dd548b688a7adba2f8068cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb10d64dd548b688a7adba2f8068cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41391 } } }
I20260926 04:11:39.821658 20036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:39.821709 20034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31eb10d64dd548b688a7adba2f8068cd. Latest consensus state: current_term: 1 leader_uuid: "31eb10d64dd548b688a7adba2f8068cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb10d64dd548b688a7adba2f8068cd" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 41391 } } }
I20260926 04:11:39.821791 20034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:39.821868 20040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:39.822438 20040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:39.824065 20040 catalog_manager.cc:1347] Generated new cluster ID: 61463a7fa29f45c8b65a23008f3f15dc
I20260926 04:11:39.824141 20040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:11:39 dist-test-slave-xf24 krb5kdc[19948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:39.833742 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:39.820993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.213.254:41391)
0926 04:11:39.822079 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:39.822087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:39.822123 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:11:39.822252 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:39.822261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:39.822323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:39.822463 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:39.823441 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:39.824808 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:39.825875 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:39.826544 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:39.826663 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:39.829721 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:39.829726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:39.829726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:39.829742 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:39.831892 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:39.832395 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:39.832398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:39.832398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:39.832526 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:39.832747 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:39.832748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:39.832748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:39.833066 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:39.833220 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:39.833351 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 04:11:39.833492 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":66,"threads_started":1}
I20260926 04:11:39.843914 20040 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:39.844537 20040 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:39.850855 20040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd: Generated new TSK 0
I20260926 04:11:39.851107 20040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
Entry for principal kudu/127.0.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab.
I20260926 04:11:39.879021 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41391
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:39.985808 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:39.986025 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:39.986052 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:39.987287 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:39.987341 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:39.987368 20055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:39.988929 20055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.193.keytab
--principal=kudu/127.0.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:39.989180 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:39.989413 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:39.994700 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:39.994745 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:39.994867 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:39.995025 20055 server_base.cc:1034] running on GCE node
I20260926 04:11:39.995267 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:39.995507 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:39.996717 20055 hybrid_clock.cc:648] HybridClock initialized: now 1790395899996701 us; error 49 us; skew 500 ppm
Sep 26 04:11:39 dist-test-slave-xf24 krb5kdc[19948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:39.999325 20055 init.cc:377] Logged in from keytab as kudu/127.0.213.193@KRBTEST.COM (short username kudu)
I20260926 04:11:39.999859 20055 webserver.cc:460] Webserver started at http://127.0.213.193:36313/ using document root <none> and password file <none>
I20260926 04:11:40.000138 20055 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:40.000205 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:40.000356 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:40.001420 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "c409873934fa4f3697cb9ee3e96df0e4"
format_stamp: "Formatted at 2026-09-26 04:11:39 on dist-test-slave-xf24"
server_key: "01abbf950fa7d93d972c373fa7a337f0"
server_key_iv: "47e1b16368a433e9476133df5285a22e"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.001788 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "c409873934fa4f3697cb9ee3e96df0e4"
format_stamp: "Formatted at 2026-09-26 04:11:39 on dist-test-slave-xf24"
server_key: "01abbf950fa7d93d972c373fa7a337f0"
server_key_iv: "47e1b16368a433e9476133df5285a22e"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.003127 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:40.003872 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.004146 20055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:40.004259 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "c409873934fa4f3697cb9ee3e96df0e4"
format_stamp: "Formatted at 2026-09-26 04:11:39 on dist-test-slave-xf24"
server_key: "01abbf950fa7d93d972c373fa7a337f0"
server_key_iv: "47e1b16368a433e9476133df5285a22e"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.004359 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:40.013355 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:40.015617 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:40.015748 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:40.015951 20055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:40.016336 20055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:40.016376 20055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.016400 20055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:40.016413 20055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.023029 20055 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:37049
I20260926 04:11:40.023100 20183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:37049 every 8 connection(s)
I20260926 04:11:40.023407 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:11:40.024762 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20055
I20260926 04:11:40.024852 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:11:40.025058 855 external_mini_cluster.cc:1428] Setting key 2b8195bf258df317bd061d158d891dda
Sep 26 04:11:40 dist-test-slave-xf24 krb5kdc[19948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.193@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.194@KRBTEST.COM; defaulting to no policy
I20260926 04:11:40.034595 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.024245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:50515 (local address 127.0.213.254:41391)
0926 04:11:40.024312 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.024313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.024480 (+ 167us) server_negotiation.cc:407] Connection header received
0926 04:11:40.025623 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.025624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.025635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.025689 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.026607 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.027216 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.028287 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.028705 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.028785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.031311 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.031312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.031312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:40.031321 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.033019 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.033460 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.033461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.033461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.033537 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.033804 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.033805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.033805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.034015 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.034096 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.034434 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.034525 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 04:11:40.034762 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.024146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41391 (local address 127.0.213.193:50515)
0926 04:11:40.024374 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.024411 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.025525 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.025808 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.025814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.026227 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:40.026518 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.026524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.027318 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.027319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.028209 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.028212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.028797 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.028799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.028909 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.029508 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:40.029516 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:40.031209 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:40.033141 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:40.033147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:40.033149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:40.033379 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:40.033619 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:40.033620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:40.033620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:40.033734 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:40.034109 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:40.034115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:40.034245 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.034380 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.034547 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
I20260926 04:11:40.035327 20184 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41391
I20260926 04:11:40.035434 20184 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:40.035624 20184 heartbeater.cc:511] Master 127.0.213.254:41391 requested a full tablet report, sending...
I20260926 04:11:40.036159 19993 ts_manager.cc:194] Registered new tserver with Master: c409873934fa4f3697cb9ee3e96df0e4 (127.0.213.193:37049)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.194@KRBTEST.COM" created.
I20260926 04:11:40.036826 19993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.193@KRBTEST.COM'} at 127.0.213.193:50515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
Entry for principal kudu/127.0.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab.
I20260926 04:11:40.052016 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41391
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:40.140208 20189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:40.140462 20189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:40.140493 20189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:40.141839 20189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:40.141880 20189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:40.141911 20189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:40.143612 20189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.194.keytab
--principal=kudu/127.0.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:40.143826 20189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:40.144025 20189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:40.149822 20195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.149806 20194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:40.149906 20189 server_base.cc:1034] running on GCE node
W20260926 04:11:40.149806 20197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:40.150285 20189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:40.150625 20189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:40.151823 20189 hybrid_clock.cc:648] HybridClock initialized: now 1790395900151799 us; error 43 us; skew 500 ppm
Sep 26 04:11:40 dist-test-slave-xf24 krb5kdc[19948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:40.154770 20189 init.cc:377] Logged in from keytab as kudu/127.0.213.194@KRBTEST.COM (short username kudu)
I20260926 04:11:40.155274 20189 webserver.cc:460] Webserver started at http://127.0.213.194:38343/ using document root <none> and password file <none>
I20260926 04:11:40.155519 20189 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:40.155589 20189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:40.155747 20189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:40.156929 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "19d2bc7d559d410a84071215332db790"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "a877104efd304f9a31f9e46f1090e037"
server_key_iv: "88ed001d84ad4d0f2d58eab76c47e287"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.157315 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "19d2bc7d559d410a84071215332db790"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "a877104efd304f9a31f9e46f1090e037"
server_key_iv: "88ed001d84ad4d0f2d58eab76c47e287"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.158752 20189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:11:40.159727 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.160002 20189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:40.160148 20189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "19d2bc7d559d410a84071215332db790"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "a877104efd304f9a31f9e46f1090e037"
server_key_iv: "88ed001d84ad4d0f2d58eab76c47e287"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.160243 20189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:40.173369 20189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:40.175585 20189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:40.175774 20189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:40.176049 20189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:40.176484 20189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:40.176550 20189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.176604 20189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:40.176636 20189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.183056 20189 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:36923
I20260926 04:11:40.183139 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:36923 every 8 connection(s)
I20260926 04:11:40.183497 20189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:40.187438 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20189
I20260926 04:11:40.187657 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:11:40.187945 855 external_mini_cluster.cc:1428] Setting key 825d3a64d71a65b01bd3ce453abaca1d
Sep 26 04:11:40 dist-test-slave-xf24 krb5kdc[19948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.194@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:40.195369 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.184475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:40199 (local address 127.0.213.254:41391)
0926 04:11:40.184564 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.184567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.184746 (+ 179us) server_negotiation.cc:407] Connection header received
0926 04:11:40.185661 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.185662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.185672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.185735 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.186439 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.187004 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.188533 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.188832 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.188896 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.191783 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.191785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.191785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:40.191793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.193691 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.194212 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.194214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.194214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.194280 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.194565 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.194567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.194567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.194734 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.194825 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.195156 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.195267 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:11:40.195591 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.184399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41391 (local address 127.0.213.194:40199)
0926 04:11:40.184582 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.184622 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.185545 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.185792 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.185795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.186085 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:40.186343 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.186348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.187117 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.187118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.188397 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.188401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.188940 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.188942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.189091 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.189728 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:40.189737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:40.191644 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:40.193820 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:40.193831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:40.193833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:40.194106 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:40.194367 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:40.194368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:40.194368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:40.194492 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:40.194815 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:40.194823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:40.194988 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.195132 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.195314 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260926 04:11:40.196244 20318 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41391
I20260926 04:11:40.196377 20318 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:40.196573 20318 heartbeater.cc:511] Master 127.0.213.254:41391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.213.195@KRBTEST.COM; defaulting to no policy
I20260926 04:11:40.197013 19993 ts_manager.cc:194] Registered new tserver with Master: 19d2bc7d559d410a84071215332db790 (127.0.213.194:36923)
I20260926 04:11:40.197420 19993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.194@KRBTEST.COM'} at 127.0.213.194:40199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
Entry for principal kudu/127.0.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab.
I20260926 04:11:40.215654 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41391
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:40.299422 20323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:40.299674 20323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:40.299701 20323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:40.300945 20323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:40.300987 20323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:40.301014 20323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:40.302505 20323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/krb5kdc/kudu_127.0.213.195.keytab
--principal=kudu/127.0.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:40.302717 20323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:40.302901 20323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:40.308135 20328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.308194 20329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.308167 20331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:40.308233 20323 server_base.cc:1034] running on GCE node
I20260926 04:11:40.308570 20323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:40.308820 20323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:40.309993 20323 hybrid_clock.cc:648] HybridClock initialized: now 1790395900309979 us; error 43 us; skew 500 ppm
Sep 26 04:11:40 dist-test-slave-xf24 krb5kdc[19948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:40.312724 20323 init.cc:377] Logged in from keytab as kudu/127.0.213.195@KRBTEST.COM (short username kudu)
I20260926 04:11:40.313161 20323 webserver.cc:460] Webserver started at http://127.0.213.195:39075/ using document root <none> and password file <none>
I20260926 04:11:40.313349 20323 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:40.313390 20323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:40.313485 20323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:40.314473 20323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "390ca83455da43869ca5ec0191c9550d"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "b6de1dce33549c067f05e241b6ee2b3f"
server_key_iv: "10495cc35ec910f4bf45450664745ab3"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.314783 20323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "390ca83455da43869ca5ec0191c9550d"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "b6de1dce33549c067f05e241b6ee2b3f"
server_key_iv: "10495cc35ec910f4bf45450664745ab3"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.316037 20323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:11:40.316905 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.317060 20323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:11:40.317158 20323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "390ca83455da43869ca5ec0191c9550d"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "b6de1dce33549c067f05e241b6ee2b3f"
server_key_iv: "10495cc35ec910f4bf45450664745ab3"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.317216 20323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:40.325603 20323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:40.327762 20323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:40.327883 20323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:40.328147 20323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:40.328474 20323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:40.328506 20323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.328552 20323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:40.328578 20323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.334654 20323 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:41423
I20260926 04:11:40.334718 20451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:41423 every 8 connection(s)
I20260926 04:11:40.335139 20323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:11:40.341044 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20323
I20260926 04:11:40.341158 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:11:40.341325 855 external_mini_cluster.cc:1428] Setting key 9cf437e4197eb62c552fc86b9cc40115
Sep 26 04:11:40 dist-test-slave-xf24 krb5kdc[19948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.195@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:40.345589 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.335980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:34831 (local address 127.0.213.254:41391)
0926 04:11:40.336048 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.336051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.336269 (+ 218us) server_negotiation.cc:407] Connection header received
0926 04:11:40.337344 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.337345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.337359 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.337419 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.338184 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.338711 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.339766 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.339965 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.340047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.342745 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.342748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.342748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:40.342756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.344119 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.344581 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.344582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.344582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.344646 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.344903 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.344904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.344905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.345037 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.345115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.345409 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.345491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:11:40.345795 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.335929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:41391 (local address 127.0.213.195:34831)
0926 04:11:40.336137 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.336176 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.337204 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.337508 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.337512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.337857 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:40.338074 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.338080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.338820 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.338822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.339665 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.339667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.340090 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.340091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.340219 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.340795 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:11:40.340805 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:40.342643 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:40.344197 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:40.344204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:40.344206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:40.344442 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:40.344720 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:11:40.344721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:11:40.344721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:11:40.344831 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:11:40.345124 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:40.345131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:40.345270 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.345375 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.345555 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260926 04:11:40.346292 20452 heartbeater.cc:347] Connected to a master server at 127.0.213.254:41391
I20260926 04:11:40.346386 20452 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:40.346534 20452 heartbeater.cc:511] Master 127.0.213.254:41391 requested a full tablet report, sending...
I20260926 04:11:40.346928 19993 ts_manager.cc:194] Registered new tserver with Master: 390ca83455da43869ca5ec0191c9550d (127.0.213.195:41423)
I20260926 04:11:40.347265 19993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.213.195@KRBTEST.COM'} at 127.0.213.195:34831
I20260926 04:11:40.353616 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:11:40.361991 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.355424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.213.254:41391)
0926 04:11:40.355507 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.355509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.355642 (+ 133us) server_negotiation.cc:407] Connection header received
0926 04:11:40.355746 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.355747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.355757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.355832 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.356487 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.357007 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.357771 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.357965 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.358042 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.359296 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.359297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.359298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:40.359305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.360729 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.361113 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.361114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.361115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.361173 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:40.361376 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:40.361377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:40.361378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:40.361536 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.361653 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.361797 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.361874 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:11:40.364169 19993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:11:40.365863 19993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:11:40.372516 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.368457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:41423 (local address 127.0.0.1:52832)
0926 04:11:40.368834 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.368845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.368885 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.369089 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.369092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.369095 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:40.369233 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.369236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.370171 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.370172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.371340 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.371343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.372212 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.372213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.372295 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.372309 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.372386 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.372435 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":150,"threads_started":1}
I20260926 04:11:40.372543 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.368744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.213.195:41423)
0926 04:11:40.368895 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.368900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.368909 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:40.368954 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.368955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.368968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.369083 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:40.369320 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.370064 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.371431 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.372129 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.372204 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.372260 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.372413 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.372458 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260926 04:11:40.372830 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.368452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:36923 (local address 127.0.0.1:57262)
0926 04:11:40.368647 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.368673 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.368735 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.369251 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.369251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.369253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:40.369322 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.369325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.370682 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.370683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.371646 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.371648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.372605 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.372607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.372682 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.372686 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.372720 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.372759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":10,"spinlock_wait_cycles":896,"thread_start_us":86,"threads_started":1}
I20260926 04:11:40.373143 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.368692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.0.213.194:36923)
0926 04:11:40.369083 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.369088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.369097 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:40.369131 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.369133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.369146 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.369403 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:40.369487 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.370574 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.371725 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.372503 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.372941 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.373008 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.373035 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.373072 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":68,"threads_started":1}
I20260926 04:11:40.373287 20386 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab8a580b6724516b9b31bb941d3dde2 (DEFAULT_TABLE table=test-table [id=ff2d6f412aaf4af9921880dce3510cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:40.373726 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab8a580b6724516b9b31bb941d3dde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:40.376647 20469 tablet_bootstrap.cc:492] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Bootstrap starting.
I20260926 04:11:40.376885 20252 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab8a580b6724516b9b31bb941d3dde2 (DEFAULT_TABLE table=test-table [id=ff2d6f412aaf4af9921880dce3510cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:40.377290 20252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab8a580b6724516b9b31bb941d3dde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:40.377498 20469 tablet_bootstrap.cc:654] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:40.377782 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.368449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:37049 (local address 127.0.0.1:44548)
0926 04:11:40.368622 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.368663 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.368733 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.369246 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.369247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.369249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:40.369322 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.369324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.371713 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.371713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.372619 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.372622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.377442 (+ 4820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.377443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.377539 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.377546 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.377610 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.377677 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":7,"thread_start_us":61,"threads_started":1}
I20260926 04:11:40.378152 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.368875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.213.193:37049)
0926 04:11:40.369063 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.369068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.369078 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:11:40.369121 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.369123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.369138 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.369398 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:40.369477 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.370569 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.376386 (+ 5817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.377280 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.377371 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.377445 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.378010 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.378068 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260926 04:11:40.378130 20469 log.cc:824] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:40.378854 20118 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab8a580b6724516b9b31bb941d3dde2 (DEFAULT_TABLE table=test-table [id=ff2d6f412aaf4af9921880dce3510cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:40.379076 20469 tablet_bootstrap.cc:492] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: No bootstrap required, opened a new log
I20260926 04:11:40.379125 20469 ts_tablet_manager.cc:1397] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:11:40.379195 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab8a580b6724516b9b31bb941d3dde2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:40.380682 20471 tablet_bootstrap.cc:492] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Bootstrap starting.
I20260926 04:11:40.380795 20469 raft_consensus.cc:348] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.380970 20469 raft_consensus.cc:374] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:40.381006 20469 raft_consensus.cc:729] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390ca83455da43869ca5ec0191c9550d, State: Initialized, Role: FOLLOWER
I20260926 04:11:40.381130 20469 consensus_queue.cc:260] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.381342 20471 tablet_bootstrap.cc:654] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:40.381400 20469 ts_tablet_manager.cc:1428] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:11:40.381422 20452 heartbeater.cc:503] Master 127.0.213.254:41391 was elected leader, sending a full tablet report...
I20260926 04:11:40.381886 20471 log.cc:824] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:40.382452 20474 tablet_bootstrap.cc:492] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4: Bootstrap starting.
I20260926 04:11:40.382752 20471 tablet_bootstrap.cc:492] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: No bootstrap required, opened a new log
I20260926 04:11:40.382818 20471 ts_tablet_manager.cc:1397] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:40.383122 20474 tablet_bootstrap.cc:654] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:40.383698 20474 log.cc:824] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:40.384553 20474 tablet_bootstrap.cc:492] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4: No bootstrap required, opened a new log
I20260926 04:11:40.384631 20474 ts_tablet_manager.cc:1397] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:11:40.385059 20471 raft_consensus.cc:348] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.385251 20471 raft_consensus.cc:374] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:40.385285 20471 raft_consensus.cc:729] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d2bc7d559d410a84071215332db790, State: Initialized, Role: FOLLOWER
I20260926 04:11:40.385393 20471 consensus_queue.cc:260] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.385682 20471 ts_tablet_manager.cc:1428] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:40.385682 20318 heartbeater.cc:503] Master 127.0.213.254:41391 was elected leader, sending a full tablet report...
I20260926 04:11:40.386435 20474 raft_consensus.cc:348] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.386565 20474 raft_consensus.cc:374] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:40.386623 20474 raft_consensus.cc:729] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c409873934fa4f3697cb9ee3e96df0e4, State: Initialized, Role: FOLLOWER
I20260926 04:11:40.386734 20474 consensus_queue.cc:260] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.386986 20474 ts_tablet_manager.cc:1428] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:11:40.387027 20184 heartbeater.cc:503] Master 127.0.213.254:41391 was elected leader, sending a full tablet report...
W20260926 04:11:40.434252 20319 tablet.cc:2369] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:40.448988 20472 raft_consensus.cc:482] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:40.449190 20472 raft_consensus.cc:504] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.449592 20472 leader_election.cc:288] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c409873934fa4f3697cb9ee3e96df0e4 (127.0.213.193:37049), 19d2bc7d559d410a84071215332db790 (127.0.213.194:36923)
I20260926 04:11:40.453037 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.449917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:55717 (local address 127.0.213.193:37049)
0926 04:11:40.449976 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.449978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.450161 (+ 183us) server_negotiation.cc:407] Connection header received
0926 04:11:40.450194 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.450195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.450209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.450256 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:40.450419 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.451168 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.452236 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.452639 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.452698 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.452705 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.452900 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.452948 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 04:11:40.453050 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.449817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:36923 (local address 127.0.213.195:43877)
0926 04:11:40.449891 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.449907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.449978 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.450136 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.450137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.450139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:40.450219 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.450221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.451275 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.451277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.452172 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.452174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.452789 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.452790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.452851 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.452853 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.452929 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.452966 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 04:11:40.453061 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.449922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:43877 (local address 127.0.213.194:36923)
0926 04:11:40.449990 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.449992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.450004 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:11:40.450038 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.450040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.450058 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.450110 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:40.450323 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.451173 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.452263 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.452663 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.452767 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.452777 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.452950 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.452998 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:11:40.453058 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.449831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:37049 (local address 127.0.213.195:55717)
0926 04:11:40.450052 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.450063 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.450147 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.450279 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.450280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.450282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:40.450342 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.450343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.451268 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.451270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.452146 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.452149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.452735 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.452736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.452818 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.452821 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.452875 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.452930 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":130,"threads_started":1}
I20260926 04:11:40.453439 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab8a580b6724516b9b31bb941d3dde2" candidate_uuid: "390ca83455da43869ca5ec0191c9550d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d2bc7d559d410a84071215332db790" is_pre_election: true
I20260926 04:11:40.453430 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab8a580b6724516b9b31bb941d3dde2" candidate_uuid: "390ca83455da43869ca5ec0191c9550d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c409873934fa4f3697cb9ee3e96df0e4" is_pre_election: true
I20260926 04:11:40.453606 20138 raft_consensus.cc:2454] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390ca83455da43869ca5ec0191c9550d in term 0.
I20260926 04:11:40.453606 20272 raft_consensus.cc:2454] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390ca83455da43869ca5ec0191c9550d in term 0.
I20260926 04:11:40.453837 20342 leader_election.cc:302] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390ca83455da43869ca5ec0191c9550d, c409873934fa4f3697cb9ee3e96df0e4; no voters:
I20260926 04:11:40.454064 20472 raft_consensus.cc:2781] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:11:40.454123 20472 raft_consensus.cc:482] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:11:40.454149 20472 raft_consensus.cc:3037] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:40.454871 20472 raft_consensus.cc:504] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.455041 20472 leader_election.cc:288] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [CANDIDATE]: Term 1 election: Requested vote from peers c409873934fa4f3697cb9ee3e96df0e4 (127.0.213.193:37049), 19d2bc7d559d410a84071215332db790 (127.0.213.194:36923)
I20260926 04:11:40.455227 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab8a580b6724516b9b31bb941d3dde2" candidate_uuid: "390ca83455da43869ca5ec0191c9550d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d2bc7d559d410a84071215332db790"
I20260926 04:11:40.455260 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab8a580b6724516b9b31bb941d3dde2" candidate_uuid: "390ca83455da43869ca5ec0191c9550d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c409873934fa4f3697cb9ee3e96df0e4"
I20260926 04:11:40.455322 20272 raft_consensus.cc:3037] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:40.455363 20138 raft_consensus.cc:3037] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:40.456117 20272 raft_consensus.cc:2454] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390ca83455da43869ca5ec0191c9550d in term 1.
I20260926 04:11:40.456156 20138 raft_consensus.cc:2454] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390ca83455da43869ca5ec0191c9550d in term 1.
I20260926 04:11:40.456367 20339 leader_election.cc:302] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d2bc7d559d410a84071215332db790, 390ca83455da43869ca5ec0191c9550d; no voters:
I20260926 04:11:40.456501 20472 raft_consensus.cc:2781] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:40.456672 20472 raft_consensus.cc:686] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 1 LEADER]: Becoming Leader. State: Replica: 390ca83455da43869ca5ec0191c9550d, State: Running, Role: LEADER
I20260926 04:11:40.456787 20472 consensus_queue.cc:237] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } }
I20260926 04:11:40.457908 19993 catalog_manager.cc:5495] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d reported cstate change: term changed from 0 to 1, leader changed from <none> to 390ca83455da43869ca5ec0191c9550d (127.0.213.195). New cstate: current_term: 1 leader_uuid: "390ca83455da43869ca5ec0191c9550d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "390ca83455da43869ca5ec0191c9550d" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 41423 } health_report { overall_health: HEALTHY } } }
I20260926 04:11:40.464741 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.462308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.0.213.195:41423)
0926 04:11:40.462352 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.462354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.462364 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:11:40.462417 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.462418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.462434 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.462496 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:11:40.462715 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.463184 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.464027 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.464276 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.464335 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.464460 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:11:40.464555 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:11:40.464612 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.464625 (+ 13us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.464668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:11:40.466678 20272 raft_consensus.cc:1261] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 1 FOLLOWER]: Refusing update from remote peer 390ca83455da43869ca5ec0191c9550d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:40.466702 20138 raft_consensus.cc:1261] T 8ab8a580b6724516b9b31bb941d3dde2 P c409873934fa4f3697cb9ee3e96df0e4 [term 1 FOLLOWER]: Refusing update from remote peer 390ca83455da43869ca5ec0191c9550d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:40.467072 20479 consensus_queue.cc:1035] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c409873934fa4f3697cb9ee3e96df0e4" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:40.467204 20472 consensus_queue.cc:1035] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d2bc7d559d410a84071215332db790" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:40.469568 20482 mvcc.cc:205] Tried to move back new op lower bound from 7333461608308649984 to 7333461608272023552. Current Snapshot: MvccSnapshot[applied={T|T < 7333461608308649984}]
I20260926 04:11:40.469756 20483 mvcc.cc:205] Tried to move back new op lower bound from 7333461608308649984 to 7333461608272023552. Current Snapshot: MvccSnapshot[applied={T|T < 7333461608308649984}]
I20260926 04:11:40.470119 20484 mvcc.cc:205] Tried to move back new op lower bound from 7333461608308649984 to 7333461608272023552. Current Snapshot: MvccSnapshot[applied={T|T < 7333461608308649984}]
I20260926 04:11:40.472967 19993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:40.473064 19993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:40.473960 19993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31eb10d64dd548b688a7adba2f8068cd: Sending DeleteTablet for 3 replicas of tablet 8ab8a580b6724516b9b31bb941d3dde2
I20260926 04:11:40.474331 20386 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab8a580b6724516b9b31bb941d3dde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:52832
I20260926 04:11:40.474368 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab8a580b6724516b9b31bb941d3dde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:44548
I20260926 04:11:40.474493 20252 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab8a580b6724516b9b31bb941d3dde2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:40 UTC) from {username='kudu', principal='kudu/127.0.213.254@KRBTEST.COM'} at 127.0.0.1:57262
I20260926 04:11:40.474777 20494 tablet_replica.cc:330] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: stopping tablet replica
I20260926 04:11:40.474817 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20055
I20260926 04:11:40.474956 20494 raft_consensus.cc:2229] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:40.475128 20494 raft_consensus.cc:2258] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:40.475631 20494 ts_tablet_manager.cc:1905] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:40.477790 20494 ts_tablet_manager.cc:1918] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:40.477912 20494 log.cc:1196] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-1/wal/wals/8ab8a580b6724516b9b31bb941d3dde2
I20260926 04:11:40.478245 20494 ts_tablet_manager.cc:1939] T 8ab8a580b6724516b9b31bb941d3dde2 P 19d2bc7d559d410a84071215332db790: Deleting consensus metadata
I20260926 04:11:40.479022 19977 catalog_manager.cc:4841] TS 19d2bc7d559d410a84071215332db790 (127.0.213.194:36923): tablet 8ab8a580b6724516b9b31bb941d3dde2 (table test-table [id=ff2d6f412aaf4af9921880dce3510cc1]) successfully deleted
I20260926 04:11:40.479377 20493 tablet_replica.cc:330] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: stopping tablet replica
I20260926 04:11:40.479526 20493 raft_consensus.cc:2229] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:11:40.479682 20493 raft_consensus.cc:2258] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:40.480273 20493 ts_tablet_manager.cc:1905] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:40.482306 20493 ts_tablet_manager.cc:1918] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:40.482451 20493 log.cc:1196] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/8ab8a580b6724516b9b31bb941d3dde2
I20260926 04:11:40.482784 20493 ts_tablet_manager.cc:1939] T 8ab8a580b6724516b9b31bb941d3dde2 P 390ca83455da43869ca5ec0191c9550d: Deleting consensus metadata
I20260926 04:11:40.483606 19979 catalog_manager.cc:4841] TS 390ca83455da43869ca5ec0191c9550d (127.0.213.195:41423): tablet 8ab8a580b6724516b9b31bb941d3dde2 (table test-table [id=ff2d6f412aaf4af9921880dce3510cc1]) successfully deleted
W20260926 04:11:40.483997 19980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:37049 (error 108)
I20260926 04:11:40.484647 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.484490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:37049 (local address 127.0.0.1:44560)
0926 04:11:40.484565 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.484611 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 04:11:40.484709 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20189
W20260926 04:11:40.488263 19980 catalog_manager.cc:4584] TS c409873934fa4f3697cb9ee3e96df0e4 (127.0.213.193:37049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ab8a580b6724516b9b31bb941d3dde2: Network error: Client connection negotiation failed: client connection to 127.0.213.193:37049: connect: Connection refused (error 111)
I20260926 04:11:40.491036 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20323
I20260926 04:11:40.495654 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 19961
2026-09-26T04:11:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1981 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T04:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:40Z Disabled control of system clock
I20260926 04:11:40.524364 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35781
--webserver_interface=127.0.213.254
--webserver_port=0
--builtin_ntp_servers=127.0.213.212:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:40.607028 20499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:40.607268 20499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:40.607290 20499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:40.608548 20499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:40.608588 20499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:40.608601 20499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:40.608614 20499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:40.608626 20499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:40.609954 20499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:35781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:40.610160 20499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:40.610410 20499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:40.616184 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.616209 20507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.616250 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:40.616268 20499 server_base.cc:1034] running on GCE node
I20260926 04:11:40.616653 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:40.616981 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:40.618181 20499 hybrid_clock.cc:648] HybridClock initialized: now 1790395900618161 us; error 58 us; skew 500 ppm
I20260926 04:11:40.619661 20499 webserver.cc:460] Webserver started at http://127.0.213.254:35559/ using document root <none> and password file <none>
I20260926 04:11:40.619930 20499 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:40.620019 20499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:40.620219 20499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:40.621326 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "64f9d383aed791a06f9a593cf31d8b8a"
server_key_iv: "1bc29fce3861ab3f4de9929f311bc2eb"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.621701 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "64f9d383aed791a06f9a593cf31d8b8a"
server_key_iv: "1bc29fce3861ab3f4de9929f311bc2eb"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.623059 20499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:40.623893 20513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.624207 20499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:40.624316 20499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "64f9d383aed791a06f9a593cf31d8b8a"
server_key_iv: "1bc29fce3861ab3f4de9929f311bc2eb"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.624403 20499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:40.635222 20499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:40.635561 20499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:40.635715 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:40.639778 20499 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:35781
I20260926 04:11:40.639812 20565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:35781 every 8 connection(s)
I20260926 04:11:40.640216 20499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:40.641247 20566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:40.643577 20566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: Bootstrap starting.
I20260926 04:11:40.644212 20566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:40.644522 20566 log.cc:824] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:40.645344 20566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: No bootstrap required, opened a new log
I20260926 04:11:40.646998 20566 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35781 } }
I20260926 04:11:40.647151 20566 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:40.647176 20566 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07cdbbd2ad3431aac0eb7d598fe4e21, State: Initialized, Role: FOLLOWER
I20260926 04:11:40.647337 20566 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35781 } }
I20260926 04:11:40.647410 20566 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:40.647436 20566 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:40.647471 20566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:40.648144 20566 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35781 } }
I20260926 04:11:40.648252 20566 leader_election.cc:302] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07cdbbd2ad3431aac0eb7d598fe4e21; no voters:
I20260926 04:11:40.648442 20566 leader_election.cc:288] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:40.648540 20569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:40.648747 20566 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:40.648797 20569 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [term 1 LEADER]: Becoming Leader. State: Replica: a07cdbbd2ad3431aac0eb7d598fe4e21, State: Running, Role: LEADER
I20260926 04:11:40.648897 20569 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35781 } }
I20260926 04:11:40.648989 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20499
I20260926 04:11:40.649080 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:40.649283 855 external_mini_cluster.cc:1428] Setting key 4ed3f9a984fdbb8a45b07316d937a1a0
I20260926 04:11:40.649505 20571 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07cdbbd2ad3431aac0eb7d598fe4e21. Latest consensus state: current_term: 1 leader_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35781 } } }
I20260926 04:11:40.649593 20571 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:40.650041 20570 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07cdbbd2ad3431aac0eb7d598fe4e21" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35781 } } }
I20260926 04:11:40.650175 20570 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21 [sys.catalog]: This master's current role is: LEADER
W20260926 04:11:40.650985 20587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:11:40.651052 20587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:11:40.651100 20578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:40.651257 20578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:40.652916 20578 catalog_manager.cc:1347] Generated new cluster ID: 29cc10892fda407ba308da3b6602a8f6
I20260926 04:11:40.652979 20578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:40.659658 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.649974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.0.213.254:35781)
0926 04:11:40.650173 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.650183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.650228 (+ 45us) server_negotiation.cc:407] Connection header received
0926 04:11:40.650357 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.650364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.650545 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.650719 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.651012 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.652274 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.656790 (+ 4516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.657757 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.657942 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.658793 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.658799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.658799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:40.659063 (+ 264us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.659117 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:40.659136 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.659220 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.659290 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.659394 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260926 04:11:40.670233 20578 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:40.670984 20578 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:40.684689 20578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: Generated new TSK 0
I20260926 04:11:40.684918 20578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:40.686909 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.193:0
--local_ip_for_outbound_sockets=127.0.213.193
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35781
--builtin_ntp_servers=127.0.213.212:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:11:40.770509 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:40.770745 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:40.770769 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:40.772127 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:40.772176 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:40.772204 20590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.193
I20260926 04:11:40.773633 20590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.213.193
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.193
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:40.773891 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:40.774223 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:40.781064 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.781116 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.781255 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:40.781585 20590 server_base.cc:1034] running on GCE node
I20260926 04:11:40.781804 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:40.782092 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:40.783255 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790395900783238 us; error 32 us; skew 500 ppm
I20260926 04:11:40.785053 20590 webserver.cc:460] Webserver started at http://127.0.213.193:36905/ using document root <none> and password file <none>
I20260926 04:11:40.785365 20590 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:40.785447 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:40.785585 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:40.786988 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data/instance:
uuid: "3524a6111bdf4db3bc2a2859f6c6eddc"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "6ff2650492ca4bbe5c67efa1744332b6"
server_key_iv: "ba749e666c85d8c3acf89be0968f9ca6"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.787406 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal/instance:
uuid: "3524a6111bdf4db3bc2a2859f6c6eddc"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "6ff2650492ca4bbe5c67efa1744332b6"
server_key_iv: "ba749e666c85d8c3acf89be0968f9ca6"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.789186 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 04:11:40.790133 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.790449 20590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:40.790549 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal
uuid: "3524a6111bdf4db3bc2a2859f6c6eddc"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "6ff2650492ca4bbe5c67efa1744332b6"
server_key_iv: "ba749e666c85d8c3acf89be0968f9ca6"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.790621 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:40.808218 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:40.808524 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:40.808650 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:40.808866 20590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:40.809237 20590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:40.809273 20590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.809329 20590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:40.809356 20590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.815719 20590 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.193:43067
I20260926 04:11:40.815761 20717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.193:43067 every 8 connection(s)
I20260926 04:11:40.816326 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/data/info.pb
I20260926 04:11:40.822255 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20590
I20260926 04:11:40.822376 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-0/wal/instance
I20260926 04:11:40.822605 855 external_mini_cluster.cc:1428] Setting key 45d84f2eb8e06194764dc58b5e69189c
I20260926 04:11:40.822654 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.817297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.193:53159 (local address 127.0.213.254:35781)
0926 04:11:40.817373 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.817376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.817687 (+ 311us) server_negotiation.cc:407] Connection header received
0926 04:11:40.818204 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.818205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.818220 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.818295 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.819153 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.819706 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.820902 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.821131 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.821220 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.822207 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.822209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.822210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:40.822217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.822236 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:40.822243 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.822310 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.822515 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.822564 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 04:11:40.822839 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.817278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35781 (local address 127.0.213.193:53159)
0926 04:11:40.817543 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.817583 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.818095 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.818423 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.818427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.818437 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:40.819026 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.819032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.819822 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.819824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.820784 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.820788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.821261 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.821263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.821397 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.822021 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:40.822021 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:40.822053 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:40.822080 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:40.822316 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:40.822321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:40.822323 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.822432 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.822537 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":114,"threads_started":1}
I20260926 04:11:40.823402 20718 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35781
I20260926 04:11:40.823523 20718 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:40.823725 20718 heartbeater.cc:511] Master 127.0.213.254:35781 requested a full tablet report, sending...
I20260926 04:11:40.823730 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.194:0
--local_ip_for_outbound_sockets=127.0.213.194
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35781
--builtin_ntp_servers=127.0.213.212:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:11:40.824265 20530 ts_manager.cc:194] Registered new tserver with Master: 3524a6111bdf4db3bc2a2859f6c6eddc (127.0.213.193:43067)
I20260926 04:11:40.824986 20530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.193:53159
W20260926 04:11:40.926258 20721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:40.926550 20721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:40.926582 20721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:40.928345 20721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:40.928411 20721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:40.928450 20721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.194
I20260926 04:11:40.930462 20721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.213.194
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.194
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:40.930750 20721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:40.931114 20721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:40.937856 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.937871 20726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:40.938012 20727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:40.938309 20721 server_base.cc:1034] running on GCE node
I20260926 04:11:40.938725 20721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:40.939117 20721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:40.940285 20721 hybrid_clock.cc:648] HybridClock initialized: now 1790395900940257 us; error 42 us; skew 500 ppm
I20260926 04:11:40.941969 20721 webserver.cc:460] Webserver started at http://127.0.213.194:34669/ using document root <none> and password file <none>
I20260926 04:11:40.942306 20721 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:40.942373 20721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:40.942549 20721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:40.943878 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data/instance:
uuid: "f5f7d149d98044aa8c03628eb66200ad"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "e90bea33d98afc251e9d716cf2014d09"
server_key_iv: "927d6b68b9c0f49139d67efd96063a8e"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.944288 20721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal/instance:
uuid: "f5f7d149d98044aa8c03628eb66200ad"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "e90bea33d98afc251e9d716cf2014d09"
server_key_iv: "927d6b68b9c0f49139d67efd96063a8e"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.945725 20721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:40.946621 20735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.946857 20721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:40.946972 20721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal
uuid: "f5f7d149d98044aa8c03628eb66200ad"
format_stamp: "Formatted at 2026-09-26 04:11:40 on dist-test-slave-xf24"
server_key: "e90bea33d98afc251e9d716cf2014d09"
server_key_iv: "927d6b68b9c0f49139d67efd96063a8e"
server_key_version: "encryptionkey@0"
I20260926 04:11:40.947075 20721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:40.965844 20721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:40.966210 20721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:40.966383 20721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:40.966650 20721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:40.967025 20721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:40.967088 20721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.967139 20721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:40.967172 20721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:40.973793 20721 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.194:35073
I20260926 04:11:40.973871 20848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.194:35073 every 8 connection(s)
I20260926 04:11:40.974256 20721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/data/info.pb
I20260926 04:11:40.979712 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20721
I20260926 04:11:40.979836 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-1/wal/instance
I20260926 04:11:40.980108 855 external_mini_cluster.cc:1428] Setting key c321c019f3a0d60f34b75b46d82b6723
I20260926 04:11:40.980188 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.975194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:42573 (local address 127.0.213.254:35781)
0926 04:11:40.975270 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:40.975273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:40.975449 (+ 176us) server_negotiation.cc:407] Connection header received
0926 04:11:40.976019 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:40.976020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:40.976032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:40.976166 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:40.976925 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.977440 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.978510 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.978732 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.978805 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.979751 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:40.979752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:40.979753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:40.979759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:40.979777 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:40.979782 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:40.979855 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:40.980034 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 04:11:40.980098 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:11:40.980443 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:40.975104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35781 (local address 127.0.213.194:42573)
0926 04:11:40.975303 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:40.975346 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:40.975860 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:40.976268 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:40.976272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:40.976290 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:40.976806 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.976811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.977555 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.977556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.978397 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:40.978399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:40.978820 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:40.978822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:40.978939 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:40.979591 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:40.979591 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:40.979623 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:40.979647 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:40.979937 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:40.979942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:40.979945 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:40.980008 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:11:40.980189 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260926 04:11:40.981029 20849 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35781
I20260926 04:11:40.981140 20849 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:40.981143 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.213.195:0
--local_ip_for_outbound_sockets=127.0.213.195
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35781
--builtin_ntp_servers=127.0.213.212:42559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:11:40.981357 20849 heartbeater.cc:511] Master 127.0.213.254:35781 requested a full tablet report, sending...
I20260926 04:11:40.981789 20530 ts_manager.cc:194] Registered new tserver with Master: f5f7d149d98044aa8c03628eb66200ad (127.0.213.194:35073)
I20260926 04:11:40.982297 20530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.194:42573
W20260926 04:11:41.065289 20852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:41.065491 20852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:41.065513 20852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:41.066749 20852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:41.066794 20852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:41.066820 20852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.213.195
I20260926 04:11:41.068228 20852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.213.195
--webserver_port=0
--tserver_master_addrs=127.0.213.254:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.213.195
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:41.068419 20852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:41.068655 20852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:41.073897 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:41.073897 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:41.073935 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:41.074209 20852 server_base.cc:1034] running on GCE node
I20260926 04:11:41.074435 20852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:41.074666 20852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:41.075838 20852 hybrid_clock.cc:648] HybridClock initialized: now 1790395901075825 us; error 29 us; skew 500 ppm
I20260926 04:11:41.077250 20852 webserver.cc:460] Webserver started at http://127.0.213.195:33929/ using document root <none> and password file <none>
I20260926 04:11:41.077533 20852 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:41.077611 20852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:41.077768 20852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:41.078855 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data/instance:
uuid: "e82f895600ac4ca6b9dee79c4ff47340"
format_stamp: "Formatted at 2026-09-26 04:11:41 on dist-test-slave-xf24"
server_key: "4336d493644a9edce11e21afd9ec48a2"
server_key_iv: "ffb867f0af7f42eb7afe2f8525e6872b"
server_key_version: "encryptionkey@0"
I20260926 04:11:41.079213 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal/instance:
uuid: "e82f895600ac4ca6b9dee79c4ff47340"
format_stamp: "Formatted at 2026-09-26 04:11:41 on dist-test-slave-xf24"
server_key: "4336d493644a9edce11e21afd9ec48a2"
server_key_iv: "ffb867f0af7f42eb7afe2f8525e6872b"
server_key_version: "encryptionkey@0"
I20260926 04:11:41.080624 20852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 04:11:41.081477 20866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:41.081702 20852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:41.081796 20852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal
uuid: "e82f895600ac4ca6b9dee79c4ff47340"
format_stamp: "Formatted at 2026-09-26 04:11:41 on dist-test-slave-xf24"
server_key: "4336d493644a9edce11e21afd9ec48a2"
server_key_iv: "ffb867f0af7f42eb7afe2f8525e6872b"
server_key_version: "encryptionkey@0"
I20260926 04:11:41.081852 20852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:41.116995 20852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:41.117297 20852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:41.117472 20852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:41.117712 20852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:11:41.118132 20852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:11:41.118194 20852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:41.118230 20852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:11:41.118273 20852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:41.124326 20852 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.195:44245
I20260926 04:11:41.124369 20979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.195:44245 every 8 connection(s)
I20260926 04:11:41.124771 20852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/data/info.pb
I20260926 04:11:41.127108 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 20852
I20260926 04:11:41.127249 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal/instance
I20260926 04:11:41.127445 855 external_mini_cluster.cc:1428] Setting key 691cfeb94e60b4f6cb340b85f3c66288
I20260926 04:11:41.131170 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.125937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.195:60095 (local address 127.0.213.254:35781)
0926 04:11:41.126022 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.126024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.126239 (+ 215us) server_negotiation.cc:407] Connection header received
0926 04:11:41.126881 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.126882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.126896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.126954 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:41.127912 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.128717 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.129697 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.129918 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.130010 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.130870 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:41.130871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:41.130872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:41.130877 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:41.130894 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:41.130897 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:41.130951 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.131056 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.131089 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:11:41.131424 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.125916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:35781 (local address 127.0.213.195:60095)
0926 04:11:41.126112 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.126148 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:41.126744 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:41.127033 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:41.127036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:41.127062 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:11:41.127688 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.127694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.128797 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.128799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.129595 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.129598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.130020 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.130021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.130159 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.130685 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:11:41.130686 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:11:41.130748 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:11:41.130772 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:11:41.130963 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:11:41.130969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:11:41.130972 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:41.131060 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 04:11:41.131165 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260926 04:11:41.132040 20980 heartbeater.cc:347] Connected to a master server at 127.0.213.254:35781
I20260926 04:11:41.132246 20980 heartbeater.cc:464] Registering TS with master...
I20260926 04:11:41.132406 20980 heartbeater.cc:511] Master 127.0.213.254:35781 requested a full tablet report, sending...
I20260926 04:11:41.132781 20530 ts_manager.cc:194] Registered new tserver with Master: e82f895600ac4ca6b9dee79c4ff47340 (127.0.213.195:44245)
I20260926 04:11:41.133155 20530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.213.195:60095
I20260926 04:11:41.140128 855 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:11:41.145084 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.141711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.213.254:35781)
0926 04:11:41.141801 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.141803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.141899 (+ 96us) server_negotiation.cc:407] Connection header received
0926 04:11:41.141996 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.141997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.142010 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.142070 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:41.142260 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.142784 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.143611 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.143825 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.143902 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.144778 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:41.144780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:41.144780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:41.144785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:41.144809 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:41.144814 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:41.144878 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.144951 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.144982 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 04:11:41.146886 20530 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:11:41.148713 20530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:11:41.156081 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.151307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.0.213.195:44245)
0926 04:11:41.151791 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.151795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.151813 (+ 18us) server_negotiation.cc:407] Connection header received
0926 04:11:41.151868 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.151871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.151885 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.152163 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:41.152228 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.153099 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.154660 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.155387 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.155475 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.155626 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.155899 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.155946 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":80,"threads_started":1}
I20260926 04:11:41.156038 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.151136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:44245 (local address 127.0.0.1:52478)
0926 04:11:41.151400 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.151441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:41.151515 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:41.151992 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:41.151993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:41.151995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:41.152078 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.152080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.153188 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.153189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.154561 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.154564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.155649 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.155651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.155731 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.155745 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:11:41.155875 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 04:11:41.155953 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260926 04:11:41.156577 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.151361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43067 (local address 127.0.0.1:48668)
0926 04:11:41.151590 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.151603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:41.151656 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:41.151914 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:41.151915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:41.151916 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:41.151955 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.151957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.153306 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.153307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.154396 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.154398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.156332 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.156333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.156397 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.156402 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:11:41.156448 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 04:11:41.156497 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":92,"thread_start_us":49,"threads_started":1}
I20260926 04:11:41.156754 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.151150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:35073 (local address 127.0.0.1:52980)
0926 04:11:41.151458 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.151469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:41.151515 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:41.151850 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:41.151853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:41.151855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:41.151925 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.151927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.153407 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.153407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.154561 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.154564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.156484 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.156485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.156562 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.156565 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:41.156622 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:11:41.156667 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260926 04:11:41.156942 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.151190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.213.194:35073)
0926 04:11:41.151402 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.151408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.151597 (+ 189us) server_negotiation.cc:407] Connection header received
0926 04:11:41.151642 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.151645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.151662 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.151764 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:41.152011 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.153220 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.155681 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.156294 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.156360 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.156452 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.156800 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.156857 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260926 04:11:41.157271 20914 tablet_service.cc:1467] Processing CreateTablet for tablet edbc47271358458c99659e0bb77035e8 (DEFAULT_TABLE table=test-table [id=bf2ea2446d6d475daf9a5c4dcd8320fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:41.157680 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbc47271358458c99659e0bb77035e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:41.157658 20783 tablet_service.cc:1467] Processing CreateTablet for tablet edbc47271358458c99659e0bb77035e8 (DEFAULT_TABLE table=test-table [id=bf2ea2446d6d475daf9a5c4dcd8320fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:41.158007 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbc47271358458c99659e0bb77035e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:41.160341 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.151542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.0.213.193:43067)
0926 04:11:41.151713 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.151717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.151726 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:41.151764 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.151765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.151779 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.151850 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:41.152031 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.153229 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.154491 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.155389 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.155475 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.155580 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.160181 (+ 4601us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.160247 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260926 04:11:41.161114 20995 tablet_bootstrap.cc:492] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Bootstrap starting.
I20260926 04:11:41.161089 20652 tablet_service.cc:1467] Processing CreateTablet for tablet edbc47271358458c99659e0bb77035e8 (DEFAULT_TABLE table=test-table [id=bf2ea2446d6d475daf9a5c4dcd8320fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:11:41.161456 20652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbc47271358458c99659e0bb77035e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:41.161810 20995 tablet_bootstrap.cc:654] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:41.162151 20995 log.cc:824] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:41.163131 20995 tablet_bootstrap.cc:492] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: No bootstrap required, opened a new log
I20260926 04:11:41.163206 20995 ts_tablet_manager.cc:1397] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:41.164505 20997 tablet_bootstrap.cc:492] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: Bootstrap starting.
I20260926 04:11:41.164623 20994 tablet_bootstrap.cc:492] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Bootstrap starting.
I20260926 04:11:41.165230 20995 raft_consensus.cc:348] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.165318 20994 tablet_bootstrap.cc:654] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:41.165318 20997 tablet_bootstrap.cc:654] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:41.165386 20995 raft_consensus.cc:374] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:41.165431 20995 raft_consensus.cc:729] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f7d149d98044aa8c03628eb66200ad, State: Initialized, Role: FOLLOWER
I20260926 04:11:41.165542 20995 consensus_queue.cc:260] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.165683 20994 log.cc:824] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:41.165679 20997 log.cc:824] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:41.165836 20995 ts_tablet_manager.cc:1428] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:41.165987 20849 heartbeater.cc:503] Master 127.0.213.254:35781 was elected leader, sending a full tablet report...
I20260926 04:11:41.166532 20994 tablet_bootstrap.cc:492] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: No bootstrap required, opened a new log
I20260926 04:11:41.166615 20994 ts_tablet_manager.cc:1397] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:41.166709 20997 tablet_bootstrap.cc:492] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: No bootstrap required, opened a new log
I20260926 04:11:41.166779 20997 ts_tablet_manager.cc:1397] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:11:41.168743 20994 raft_consensus.cc:348] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.168879 20994 raft_consensus.cc:374] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:41.168780 20997 raft_consensus.cc:348] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.168902 20994 raft_consensus.cc:729] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82f895600ac4ca6b9dee79c4ff47340, State: Initialized, Role: FOLLOWER
I20260926 04:11:41.168926 20997 raft_consensus.cc:374] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:41.168977 20997 raft_consensus.cc:729] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3524a6111bdf4db3bc2a2859f6c6eddc, State: Initialized, Role: FOLLOWER
I20260926 04:11:41.169023 20994 consensus_queue.cc:260] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.169096 20997 consensus_queue.cc:260] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.169303 20994 ts_tablet_manager.cc:1428] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:41.169322 20980 heartbeater.cc:503] Master 127.0.213.254:35781 was elected leader, sending a full tablet report...
I20260926 04:11:41.169361 20997 ts_tablet_manager.cc:1428] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:11:41.169485 20718 heartbeater.cc:503] Master 127.0.213.254:35781 was elected leader, sending a full tablet report...
W20260926 04:11:41.224924 20850 tablet.cc:2369] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:41.306202 20998 raft_consensus.cc:482] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:11:41.306349 20998 raft_consensus.cc:504] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.306763 20998 leader_election.cc:288] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e82f895600ac4ca6b9dee79c4ff47340 (127.0.213.195:44245), 3524a6111bdf4db3bc2a2859f6c6eddc (127.0.213.193:43067)
I20260926 04:11:41.309741 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.307114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:56507 (local address 127.0.213.195:44245)
0926 04:11:41.307174 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.307176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.307188 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:11:41.307221 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.307222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.307229 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.307311 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:41.307478 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.307962 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.308967 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.309331 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.309450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.309459 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.309615 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.309654 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 04:11:41.309763 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.307064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.213.194:54093 (local address 127.0.213.193:43067)
0926 04:11:41.307109 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.307111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.307275 (+ 164us) server_negotiation.cc:407] Connection header received
0926 04:11:41.307337 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.307338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.307353 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.307423 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:11:41.307581 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.308043 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.309034 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.309382 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.309466 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.309473 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.309637 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.309686 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:11:41.309773 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.306968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.195:44245 (local address 127.0.213.194:56507)
0926 04:11:41.307040 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.307053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:41.307105 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:41.307318 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:41.307319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:41.307321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:41.307396 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.307398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.308089 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.308090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.308866 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.308869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.309449 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.309450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.309537 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.309541 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:11:41.309610 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 04:11:41.309656 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260926 04:11:41.309787 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.307009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43067 (local address 127.0.213.194:54093)
0926 04:11:41.307201 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.307209 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:11:41.307280 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:11:41.307428 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:11:41.307429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:11:41.307430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:11:41.307504 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.307505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.308180 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.308181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.308936 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:11:41.308938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.309465 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.309466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:11:41.309549 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.309552 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:11:41.309630 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 04:11:41.309676 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260926 04:11:41.310134 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbc47271358458c99659e0bb77035e8" candidate_uuid: "f5f7d149d98044aa8c03628eb66200ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82f895600ac4ca6b9dee79c4ff47340" is_pre_election: true
I20260926 04:11:41.310163 20672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbc47271358458c99659e0bb77035e8" candidate_uuid: "f5f7d149d98044aa8c03628eb66200ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" is_pre_election: true
I20260926 04:11:41.310303 20934 raft_consensus.cc:2454] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f7d149d98044aa8c03628eb66200ad in term 0.
I20260926 04:11:41.310325 20672 raft_consensus.cc:2454] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5f7d149d98044aa8c03628eb66200ad in term 0.
I20260926 04:11:41.310542 20738 leader_election.cc:302] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e82f895600ac4ca6b9dee79c4ff47340, f5f7d149d98044aa8c03628eb66200ad; no voters:
I20260926 04:11:41.310738 20998 raft_consensus.cc:2781] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:11:41.310778 20998 raft_consensus.cc:482] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:11:41.310796 20998 raft_consensus.cc:3037] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:41.311463 20998 raft_consensus.cc:504] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.311604 20998 leader_election.cc:288] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [CANDIDATE]: Term 1 election: Requested vote from peers e82f895600ac4ca6b9dee79c4ff47340 (127.0.213.195:44245), 3524a6111bdf4db3bc2a2859f6c6eddc (127.0.213.193:43067)
I20260926 04:11:41.311756 20672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbc47271358458c99659e0bb77035e8" candidate_uuid: "f5f7d149d98044aa8c03628eb66200ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc"
I20260926 04:11:41.311750 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbc47271358458c99659e0bb77035e8" candidate_uuid: "f5f7d149d98044aa8c03628eb66200ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82f895600ac4ca6b9dee79c4ff47340"
I20260926 04:11:41.311836 20672 raft_consensus.cc:3037] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:41.311857 20934 raft_consensus.cc:3037] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:41.312640 20672 raft_consensus.cc:2454] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f7d149d98044aa8c03628eb66200ad in term 1.
I20260926 04:11:41.312659 20934 raft_consensus.cc:2454] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f7d149d98044aa8c03628eb66200ad in term 1.
I20260926 04:11:41.312822 20736 leader_election.cc:302] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3524a6111bdf4db3bc2a2859f6c6eddc, f5f7d149d98044aa8c03628eb66200ad; no voters:
I20260926 04:11:41.312932 20998 raft_consensus.cc:2781] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:41.313069 20998 raft_consensus.cc:686] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 1 LEADER]: Becoming Leader. State: Replica: f5f7d149d98044aa8c03628eb66200ad, State: Running, Role: LEADER
I20260926 04:11:41.313185 20998 consensus_queue.cc:237] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } }
I20260926 04:11:41.314018 20530 catalog_manager.cc:5495] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f5f7d149d98044aa8c03628eb66200ad (127.0.213.194). New cstate: current_term: 1 leader_uuid: "f5f7d149d98044aa8c03628eb66200ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5f7d149d98044aa8c03628eb66200ad" member_type: VOTER last_known_addr { host: "127.0.213.194" port: 35073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 } health_report { overall_health: UNKNOWN } } }
W20260926 04:11:41.317147 20719 tablet.cc:2369] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:11:41.337982 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.335339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.0.213.194:35073)
0926 04:11:41.335423 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:41.335425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:41.335433 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:11:41.335471 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:41.335472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:41.335494 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:41.335567 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:11:41.335746 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.336262 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.337106 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:41.337312 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:41.337365 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:41.337612 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:11:41.337728 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:11:41.337829 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:41.337868 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 04:11:41.337889 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 04:11:41.340242 20934 raft_consensus.cc:1261] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 1 FOLLOWER]: Refusing update from remote peer f5f7d149d98044aa8c03628eb66200ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:41.340233 20672 raft_consensus.cc:1261] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 1 FOLLOWER]: Refusing update from remote peer f5f7d149d98044aa8c03628eb66200ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:11:41.340655 21004 consensus_queue.cc:1035] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82f895600ac4ca6b9dee79c4ff47340" member_type: VOTER last_known_addr { host: "127.0.213.195" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:41.340787 20998 consensus_queue.cc:1035] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3524a6111bdf4db3bc2a2859f6c6eddc" member_type: VOTER last_known_addr { host: "127.0.213.193" port: 43067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:11:41.343196 21007 mvcc.cc:205] Tried to move back new op lower bound from 7333461611885514752 to 7333461611779391488. Current Snapshot: MvccSnapshot[applied={T|T < 7333461611885514752}]
I20260926 04:11:41.343391 21008 mvcc.cc:205] Tried to move back new op lower bound from 7333461611885514752 to 7333461611779391488. Current Snapshot: MvccSnapshot[applied={T|T < 7333461611885514752}]
I20260926 04:11:41.344192 21009 mvcc.cc:205] Tried to move back new op lower bound from 7333461611885514752 to 7333461611779391488. Current Snapshot: MvccSnapshot[applied={T|T < 7333461611885514752}]
I20260926 04:11:41.345708 20528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:41.345825 20528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:11:41.346622 20528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a07cdbbd2ad3431aac0eb7d598fe4e21: Sending DeleteTablet for 3 replicas of tablet edbc47271358458c99659e0bb77035e8
I20260926 04:11:41.346944 20783 tablet_service.cc:1514] Processing DeleteTablet for tablet edbc47271358458c99659e0bb77035e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:41 UTC) from {username='slave'} at 127.0.0.1:52980
I20260926 04:11:41.347029 20652 tablet_service.cc:1514] Processing DeleteTablet for tablet edbc47271358458c99659e0bb77035e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:41 UTC) from {username='slave'} at 127.0.0.1:48668
I20260926 04:11:41.347023 20914 tablet_service.cc:1514] Processing DeleteTablet for tablet edbc47271358458c99659e0bb77035e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:11:41 UTC) from {username='slave'} at 127.0.0.1:52478
I20260926 04:11:41.347241 21018 tablet_replica.cc:330] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: stopping tablet replica
I20260926 04:11:41.347329 21017 tablet_replica.cc:330] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc: stopping tablet replica
I20260926 04:11:41.347340 21018 raft_consensus.cc:2229] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:41.347422 21017 raft_consensus.cc:2229] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:11:41.347440 21018 raft_consensus.cc:2258] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:41.347529 21017 raft_consensus.cc:2258] T edbc47271358458c99659e0bb77035e8 P 3524a6111bdf4db3bc2a2859f6c6eddc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:41.347574 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20590
I20260926 04:11:41.347915 21018 ts_tablet_manager.cc:1905] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:11:41.349792 21018 ts_tablet_manager.cc:1918] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:11:41.349854 21018 log.cc:1196] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Deleting WAL directory at /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395654118158-855-0/minicluster-data/ts-2/wal/wals/edbc47271358458c99659e0bb77035e8
I20260926 04:11:41.350178 21018 ts_tablet_manager.cc:1939] T edbc47271358458c99659e0bb77035e8 P e82f895600ac4ca6b9dee79c4ff47340: Deleting consensus metadata
I20260926 04:11:41.350878 20516 catalog_manager.cc:4841] TS e82f895600ac4ca6b9dee79c4ff47340 (127.0.213.195:44245): tablet edbc47271358458c99659e0bb77035e8 (table test-table [id=bf2ea2446d6d475daf9a5c4dcd8320fa]) successfully deleted
I20260926 04:11:41.352605 21016 tablet_replica.cc:330] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: stopping tablet replica
I20260926 04:11:41.352725 21016 raft_consensus.cc:2229] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:11:41.352835 21016 raft_consensus.cc:2258] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:11:41.353248 21016 ts_tablet_manager.cc:1905] T edbc47271358458c99659e0bb77035e8 P f5f7d149d98044aa8c03628eb66200ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 04:11:41.354188 20514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.213.193:43067 (error 108)
I20260926 04:11:41.354486 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20721
I20260926 04:11:41.354854 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.354726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.193:43067 (local address 127.0.0.1:48672)
0926 04:11:41.354777 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.354812 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:11:41.355109 20514 catalog_manager.cc:4584] TS 3524a6111bdf4db3bc2a2859f6c6eddc (127.0.213.193:43067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edbc47271358458c99659e0bb77035e8: Network error: Client connection negotiation failed: client connection to 127.0.213.193:43067: connect: Connection refused (error 111)
I20260926 04:11:41.359469 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20852
I20260926 04:11:41.360364 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:41.359638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.194:35073 (local address 127.0.0.1:52986)
0926 04:11:41.360297 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 04:11:41.360336 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.194:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638}
I20260926 04:11:41.364480 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 20499
2026-09-26T04:11:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (867 ms)
[----------] 6 tests from EncryptionPolicyTest (8506 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:41 dist-test-slave-xf24 krb5kdc[21022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:41 dist-test-slave-xf24 krb5kdc[21022](info): set up 2 sockets
Sep 26 04:11:41 dist-test-slave-xf24 krb5kdc[21022](info): commencing operation
krb5kdc: starting...
W20260926 04:11:42.424243 855 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:42 dist-test-slave-xf24 krb5kdc[21022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:42.552014 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:42.639585 21035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:42.639838 21035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:42.639866 21035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:42.641244 21035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:42.641301 21035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:42.641317 21035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:42.641331 21035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:42.641345 21035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:42.642990 21035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:42.643220 21035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:42.643416 21035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:42.649118 21043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:42.649083 21041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:42.649194 21035 server_base.cc:1034] running on GCE node
W20260926 04:11:42.649107 21040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:42.649677 21035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:42.649955 21035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:42.651166 21035 hybrid_clock.cc:648] HybridClock initialized: now 1790395902651139 us; error 42 us; skew 500 ppm
Sep 26 04:11:42 dist-test-slave-xf24 krb5kdc[21022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:42.659502 21035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:42.660270 21035 webserver.cc:460] Webserver started at http://127.0.0.1:38395/ using document root <none> and password file <none>
I20260926 04:11:42.660560 21035 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:42.660651 21035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:42.660826 21035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:42.662043 21035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "32c11d11a142455981ab8213c2784ee0"
format_stamp: "Formatted at 2026-09-26 04:11:42 on dist-test-slave-xf24"
server_key: "713309f17477bbd8837a2e873685a50a"
server_key_iv: "7ef451d84c80b2cb97e8252fff8d0c46"
server_key_version: "encryptionkey@0"
I20260926 04:11:42.662433 21035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "32c11d11a142455981ab8213c2784ee0"
format_stamp: "Formatted at 2026-09-26 04:11:42 on dist-test-slave-xf24"
server_key: "713309f17477bbd8837a2e873685a50a"
server_key_iv: "7ef451d84c80b2cb97e8252fff8d0c46"
server_key_version: "encryptionkey@0"
I20260926 04:11:42.664024 21035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:42.665114 21050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:42.665338 21035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:42.665448 21035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "32c11d11a142455981ab8213c2784ee0"
format_stamp: "Formatted at 2026-09-26 04:11:42 on dist-test-slave-xf24"
server_key: "713309f17477bbd8837a2e873685a50a"
server_key_iv: "7ef451d84c80b2cb97e8252fff8d0c46"
server_key_version: "encryptionkey@0"
I20260926 04:11:42.665545 21035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:42.680384 21035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:42.682901 21035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:42.683085 21035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:42.687439 21035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35729
I20260926 04:11:42.687558 21102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35729 every 8 connection(s)
I20260926 04:11:42.687894 21035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:42.688835 21103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:42.691082 21103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0: Bootstrap starting.
I20260926 04:11:42.691633 21103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:42.691869 21103 log.cc:824] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:42.692663 21103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0: No bootstrap required, opened a new log
I20260926 04:11:42.694371 21103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c11d11a142455981ab8213c2784ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260926 04:11:42.694528 21103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:42.694551 21103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c11d11a142455981ab8213c2784ee0, State: Initialized, Role: FOLLOWER
I20260926 04:11:42.694708 21103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c11d11a142455981ab8213c2784ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260926 04:11:42.694792 21103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:42.694816 21103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:42.694847 21103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:42.695408 21103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c11d11a142455981ab8213c2784ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260926 04:11:42.695499 21103 leader_election.cc:302] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c11d11a142455981ab8213c2784ee0; no voters:
I20260926 04:11:42.695672 21103 leader_election.cc:288] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:42.695724 21106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:42.695942 21106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [term 1 LEADER]: Becoming Leader. State: Replica: 32c11d11a142455981ab8213c2784ee0, State: Running, Role: LEADER
I20260926 04:11:42.696031 21103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:42.696144 21106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c11d11a142455981ab8213c2784ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } }
I20260926 04:11:42.696656 21107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c11d11a142455981ab8213c2784ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c11d11a142455981ab8213c2784ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } } }
I20260926 04:11:42.696749 21107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:42.696925 21108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c11d11a142455981ab8213c2784ee0. Latest consensus state: current_term: 1 leader_uuid: "32c11d11a142455981ab8213c2784ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c11d11a142455981ab8213c2784ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35729 } } }
I20260926 04:11:42.697054 21108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:42.697394 21115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:42.697606 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21035
I20260926 04:11:42.697742 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:42.697937 855 external_mini_cluster.cc:1428] Setting key 5b1923db5e5d91f2a95004ad1caf8f20
I20260926 04:11:42.698006 21115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:42.700147 21115 catalog_manager.cc:1347] Generated new cluster ID: e1771de3876f4bf19ad556aa8864354b
I20260926 04:11:42.700220 21115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:42.706521 21115 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:42.707147 21115 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 04:11:42 dist-test-slave-xf24 krb5kdc[21022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:11:42.710562 21115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c11d11a142455981ab8213c2784ee0: Generated new TSK 0
I20260926 04:11:42.710747 21115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:42.712347 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:42.698504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56097 (local address 127.0.0.1:35729)
0926 04:11:42.700430 (+ 1926us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:42.700440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:42.700475 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:11:42.700616 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:42.700623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:42.700681 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:42.700827 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:42.701593 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:42.703068 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:42.704162 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:42.704897 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:42.705074 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:42.708252 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:42.708257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:42.708257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:42.708272 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:42.710498 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:42.711006 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:42.711009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:42.711009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:42.711100 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:42.711421 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:42.711422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:42.711422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:42.711701 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:42.711838 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:42.711991 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 04:11:42.712146 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1874,"thread_start_us":86,"threads_started":1}
I20260926 04:11:42.713367 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:42.721119 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:42.714787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44277 (local address 127.0.0.1:35729)
0926 04:11:42.714832 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:42.714833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:42.714949 (+ 116us) server_negotiation.cc:407] Connection header received
0926 04:11:42.715046 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:42.715046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:42.715055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:42.715101 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:42.715690 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:42.716286 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:42.717031 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:42.717246 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:42.717334 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:42.718569 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:42.718570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:42.718570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:42.718577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:42.719921 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:42.720316 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:42.720317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:42.720317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:42.720374 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:42.720598 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:42.720599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:42.720599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:42.720707 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:42.720821 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:42.720949 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 04:11:42.721010 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:11:42.723030 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21035
W20260926 04:11:42.726773 21020 connection.cc:535] client connection to 127.0.0.1:35729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35729): Connection reset by peer (error 104)
2026-09-26T04:11:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:42 dist-test-slave-xf24 krb5kdc[21134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:42 dist-test-slave-xf24 krb5kdc[21134](info): set up 2 sockets
Sep 26 04:11:42 dist-test-slave-xf24 krb5kdc[21134](info): commencing operation
krb5kdc: starting...
W20260926 04:11:43.776034 855 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:43 dist-test-slave-xf24 krb5kdc[21134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:43.896745 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:43.978380 21147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:43.978791 21147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:43.978842 21147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:43.980419 21147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:43.980499 21147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:43.980515 21147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:43.980528 21147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:43.980541 21147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:43.982287 21147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:43.982533 21147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:43.982725 21147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:11:43.988009 21147 server_base.cc:1034] running on GCE node
W20260926 04:11:43.987987 21152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:43.987986 21153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:43.987987 21155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:43.988401 21147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:43.988734 21147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:43.989938 21147 hybrid_clock.cc:648] HybridClock initialized: now 1790395903989888 us; error 63 us; skew 500 ppm
Sep 26 04:11:43 dist-test-slave-xf24 krb5kdc[21134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:43.992868 21147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:43.993418 21147 webserver.cc:460] Webserver started at http://127.0.0.1:40989/ using document root <none> and password file <none>
I20260926 04:11:43.993675 21147 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:43.993752 21147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:43.993908 21147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:43.995033 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "a4ec63ba619f47d68d361aed64b07525"
format_stamp: "Formatted at 2026-09-26 04:11:43 on dist-test-slave-xf24"
server_key: "a89348ce618c4dc3e606a7467a68f82f"
server_key_iv: "935828fb7bc160c952d23e14e43b1814"
server_key_version: "encryptionkey@0"
I20260926 04:11:43.995411 21147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "a4ec63ba619f47d68d361aed64b07525"
format_stamp: "Formatted at 2026-09-26 04:11:43 on dist-test-slave-xf24"
server_key: "a89348ce618c4dc3e606a7467a68f82f"
server_key_iv: "935828fb7bc160c952d23e14e43b1814"
server_key_version: "encryptionkey@0"
I20260926 04:11:43.996991 21147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:43.997855 21162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:43.998104 21147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:43.998216 21147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "a4ec63ba619f47d68d361aed64b07525"
format_stamp: "Formatted at 2026-09-26 04:11:43 on dist-test-slave-xf24"
server_key: "a89348ce618c4dc3e606a7467a68f82f"
server_key_iv: "935828fb7bc160c952d23e14e43b1814"
server_key_version: "encryptionkey@0"
I20260926 04:11:43.998296 21147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:44.017298 21147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:44.019820 21147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:44.019959 21147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:44.024281 21147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38783
I20260926 04:11:44.024333 21214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38783 every 8 connection(s)
I20260926 04:11:44.024755 21147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:44.025714 21215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:44.028015 21215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525: Bootstrap starting.
I20260926 04:11:44.028715 21215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:44.029014 21215 log.cc:824] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:44.029795 21215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525: No bootstrap required, opened a new log
I20260926 04:11:44.031420 21215 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ec63ba619f47d68d361aed64b07525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260926 04:11:44.031567 21215 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:44.031590 21215 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ec63ba619f47d68d361aed64b07525, State: Initialized, Role: FOLLOWER
I20260926 04:11:44.031764 21215 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ec63ba619f47d68d361aed64b07525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260926 04:11:44.031831 21215 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:44.031873 21215 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:44.031924 21215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:44.032205 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21147
I20260926 04:11:44.032323 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:44.032510 855 external_mini_cluster.cc:1428] Setting key 82b962e44ba667e9cc2c8d6c5042d205
I20260926 04:11:44.032693 21215 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ec63ba619f47d68d361aed64b07525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260926 04:11:44.032826 21215 leader_election.cc:302] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ec63ba619f47d68d361aed64b07525; no voters:
I20260926 04:11:44.033026 21215 leader_election.cc:288] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:44.033079 21218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:44.033294 21218 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [term 1 LEADER]: Becoming Leader. State: Replica: a4ec63ba619f47d68d361aed64b07525, State: Running, Role: LEADER
I20260926 04:11:44.033324 21215 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:44.033466 21218 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ec63ba619f47d68d361aed64b07525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } }
I20260926 04:11:44.033919 21221 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ec63ba619f47d68d361aed64b07525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ec63ba619f47d68d361aed64b07525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } } }
I20260926 04:11:44.034006 21221 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:44.034170 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ec63ba619f47d68d361aed64b07525. Latest consensus state: current_term: 1 leader_uuid: "a4ec63ba619f47d68d361aed64b07525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ec63ba619f47d68d361aed64b07525" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38783 } } }
I20260926 04:11:44.034262 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:44.034216 21224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:44.034802 21224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:44.037175 21224 catalog_manager.cc:1347] Generated new cluster ID: 3f7918f293d847d986df7f36da947afd
I20260926 04:11:44.037236 21224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:44.041087 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:44.033273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34813 (local address 127.0.0.1:38783)
0926 04:11:44.033440 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:44.033450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:44.033485 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:11:44.033612 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:44.033619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:44.033676 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:44.033828 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:44.034114 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:44.035573 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:44.036538 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:44.037124 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:44.037232 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:44.040336 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:44.040340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:44.040341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:44.040579 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:44.040623 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:44.040640 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:44.040738 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:44.040835 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 04:11:44.040864 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260926 04:11:44.042395 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:44.048166 21224 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:44.048874 21224 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:44.051455 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:44.047625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44995 (local address 127.0.0.1:38783)
0926 04:11:44.047900 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:44.047902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:44.047911 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:11:44.047934 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:44.047935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:44.047945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:44.048007 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:44.048223 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:44.048942 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:44.050013 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:44.050325 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:44.050392 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:44.051178 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:44.051179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:44.051179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:44.051184 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:44.051200 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:44.051204 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:44.051264 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:44.051362 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 04:11:44.051374 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":240}
I20260926 04:11:44.053565 21224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ec63ba619f47d68d361aed64b07525: Generated new TSK 0
I20260926 04:11:44.053766 21224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:44.057600 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21147
W20260926 04:11:44.061772 21132 connection.cc:535] client connection to 127.0.0.1:38783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38783): Connection reset by peer (error 104)
2026-09-26T04:11:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:44 dist-test-slave-xf24 krb5kdc[21246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:44 dist-test-slave-xf24 krb5kdc[21246](info): set up 2 sockets
Sep 26 04:11:44 dist-test-slave-xf24 krb5kdc[21246](info): commencing operation
krb5kdc: starting...
W20260926 04:11:45.112335 855 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:45 dist-test-slave-xf24 krb5kdc[21246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:45.240312 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:45.322263 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:45.322475 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:45.322499 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:45.323748 21259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:45.323799 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:45.323813 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:45.323827 21259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:45.323838 21259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:45.325377 21259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:45.325632 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:45.325860 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:45.331085 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:45.331050 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:45.331113 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:45.331211 21259 server_base.cc:1034] running on GCE node
I20260926 04:11:45.331548 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:45.331861 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:45.333030 21259 hybrid_clock.cc:648] HybridClock initialized: now 1790395905333011 us; error 40 us; skew 500 ppm
Sep 26 04:11:45 dist-test-slave-xf24 krb5kdc[21246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:45.335737 21259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:45.336259 21259 webserver.cc:460] Webserver started at http://127.0.0.1:45387/ using document root <none> and password file <none>
I20260926 04:11:45.336510 21259 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:45.336578 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:45.336736 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:45.337841 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "f37dda5afdc74823969af8aea5b66e10"
format_stamp: "Formatted at 2026-09-26 04:11:45 on dist-test-slave-xf24"
server_key: "83151c30c1759fbe5cf723ebc5060abb"
server_key_iv: "720a960fa2c052f1f51ef9b2fd0d1933"
server_key_version: "encryptionkey@0"
I20260926 04:11:45.338228 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "f37dda5afdc74823969af8aea5b66e10"
format_stamp: "Formatted at 2026-09-26 04:11:45 on dist-test-slave-xf24"
server_key: "83151c30c1759fbe5cf723ebc5060abb"
server_key_iv: "720a960fa2c052f1f51ef9b2fd0d1933"
server_key_version: "encryptionkey@0"
I20260926 04:11:45.339756 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:45.340644 21274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:45.340866 21259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:11:45.340976 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "f37dda5afdc74823969af8aea5b66e10"
format_stamp: "Formatted at 2026-09-26 04:11:45 on dist-test-slave-xf24"
server_key: "83151c30c1759fbe5cf723ebc5060abb"
server_key_iv: "720a960fa2c052f1f51ef9b2fd0d1933"
server_key_version: "encryptionkey@0"
I20260926 04:11:45.341069 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:45.344640 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:45.347326 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:45.347499 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:45.351544 21259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41601
I20260926 04:11:45.351596 21326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41601 every 8 connection(s)
I20260926 04:11:45.351948 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:45.353034 21327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:45.355158 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21259
I20260926 04:11:45.355250 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:45.355293 21327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10: Bootstrap starting.
I20260926 04:11:45.355468 855 external_mini_cluster.cc:1428] Setting key a93f361aeb5fb59476dd09c1ef2c2091
I20260926 04:11:45.356033 21327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:45.356402 21327 log.cc:824] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:45.357155 21327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10: No bootstrap required, opened a new log
I20260926 04:11:45.358091 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:45.356118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50091 (local address 127.0.0.1:41601)
0926 04:11:45.356314 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:45.356321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:45.356346 (+ 25us) server_negotiation.cc:407] Connection header received
0926 04:11:45.356812 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:45.356819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:45.356846 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:45.356968 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:45.357645 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:45.357648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:45.357648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:45.357652 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:45.357688 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:45.357698 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:45.357768 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:45.357840 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 04:11:45.357896 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260926 04:11:45.359212 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:45.359851 21327 raft_consensus.cc:348] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37dda5afdc74823969af8aea5b66e10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41601 } }
I20260926 04:11:45.360076 21327 raft_consensus.cc:374] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:45.360245 21327 raft_consensus.cc:729] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37dda5afdc74823969af8aea5b66e10, State: Initialized, Role: FOLLOWER
I20260926 04:11:45.360394 21327 consensus_queue.cc:260] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37dda5afdc74823969af8aea5b66e10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41601 } }
I20260926 04:11:45.360520 21327 raft_consensus.cc:388] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:45.360630 21327 raft_consensus.cc:482] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:45.360699 21327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:45.361574 21327 raft_consensus.cc:504] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37dda5afdc74823969af8aea5b66e10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41601 } }
I20260926 04:11:45.361670 21327 leader_election.cc:302] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f37dda5afdc74823969af8aea5b66e10; no voters:
I20260926 04:11:45.361881 21327 leader_election.cc:288] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:45.361922 21337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:45.362078 21337 raft_consensus.cc:686] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [term 1 LEADER]: Becoming Leader. State: Replica: f37dda5afdc74823969af8aea5b66e10, State: Running, Role: LEADER
I20260926 04:11:45.362196 21327 sys_catalog.cc:564] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:45.362210 21337 consensus_queue.cc:237] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37dda5afdc74823969af8aea5b66e10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41601 } }
I20260926 04:11:45.362358 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:45.361030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52325 (local address 127.0.0.1:41601)
0926 04:11:45.361099 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:45.361101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:45.361207 (+ 106us) server_negotiation.cc:407] Connection header received
0926 04:11:45.361267 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:45.361268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:45.361272 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:45.361320 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:45.362028 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:45.362029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:45.362030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:45.362034 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:45.362060 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:45.362065 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:45.362112 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:45.362290 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 04:11:45.362303 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:11:45.362792 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f37dda5afdc74823969af8aea5b66e10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37dda5afdc74823969af8aea5b66e10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41601 } } }
I20260926 04:11:45.362902 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:45.362790 21339 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f37dda5afdc74823969af8aea5b66e10. Latest consensus state: current_term: 1 leader_uuid: "f37dda5afdc74823969af8aea5b66e10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37dda5afdc74823969af8aea5b66e10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41601 } } }
I20260926 04:11:45.363085 21339 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10 [sys.catalog]: This master's current role is: LEADER
W20260926 04:11:45.363943 21352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:11:45.364010 21352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:11:45.364058 21346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:45.364315 21346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:45.365767 21346 catalog_manager.cc:1347] Generated new cluster ID: 03e9479885474d8b98f4059ccb545025
I20260926 04:11:45.365816 21346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:45.370380 21346 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:45.370558 21346 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:45.377990 21346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f37dda5afdc74823969af8aea5b66e10: Generated new TSK 0
I20260926 04:11:45.378209 21346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:45.381250 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21259
2026-09-26T04:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:45 dist-test-slave-xf24 krb5kdc[21358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:45 dist-test-slave-xf24 krb5kdc[21358](info): set up 2 sockets
Sep 26 04:11:45 dist-test-slave-xf24 krb5kdc[21358](info): commencing operation
krb5kdc: starting...
W20260926 04:11:46.428385 855 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:46 dist-test-slave-xf24 krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:46.556177 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:46.640484 21371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:46.640734 21371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:46.640777 21371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:46.642035 21371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:46.642104 21371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:46.642131 21371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:46.642161 21371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:46.642189 21371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:46.643821 21371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:46.644138 21371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:46.644387 21371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:46.649528 21377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:46.649676 21371 server_base.cc:1034] running on GCE node
W20260926 04:11:46.649803 21376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:46.649850 21379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:46.650090 21371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:46.650352 21371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:46.651494 21371 hybrid_clock.cc:648] HybridClock initialized: now 1790395906651470 us; error 39 us; skew 500 ppm
Sep 26 04:11:46 dist-test-slave-xf24 krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:46.654043 21371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:46.654563 21371 webserver.cc:460] Webserver started at http://127.0.0.1:41661/ using document root <none> and password file <none>
I20260926 04:11:46.654801 21371 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:46.654868 21371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:46.655015 21371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:46.656167 21371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "02839c286a814855906131bce6522347"
format_stamp: "Formatted at 2026-09-26 04:11:46 on dist-test-slave-xf24"
server_key: "36b377797826b9e89b4c2e418bcec83d"
server_key_iv: "eac6d385f940139ccf07c159a4381726"
server_key_version: "encryptionkey@0"
I20260926 04:11:46.656513 21371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "02839c286a814855906131bce6522347"
format_stamp: "Formatted at 2026-09-26 04:11:46 on dist-test-slave-xf24"
server_key: "36b377797826b9e89b4c2e418bcec83d"
server_key_iv: "eac6d385f940139ccf07c159a4381726"
server_key_version: "encryptionkey@0"
I20260926 04:11:46.657918 21371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:11:46.658835 21386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:46.659058 21371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:46.659178 21371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "02839c286a814855906131bce6522347"
format_stamp: "Formatted at 2026-09-26 04:11:46 on dist-test-slave-xf24"
server_key: "36b377797826b9e89b4c2e418bcec83d"
server_key_iv: "eac6d385f940139ccf07c159a4381726"
server_key_version: "encryptionkey@0"
I20260926 04:11:46.659271 21371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:46.693610 21371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:46.696050 21371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:46.696266 21371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:46.700580 21371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38101
I20260926 04:11:46.700675 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38101 every 8 connection(s)
I20260926 04:11:46.701049 21371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:46.701536 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21371
I20260926 04:11:46.701663 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:46.701879 855 external_mini_cluster.cc:1428] Setting key 1c995d53520c93c2b166046ba1e4e217
I20260926 04:11:46.702275 21439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:46.705642 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347: Bootstrap starting.
I20260926 04:11:46.706411 21439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:46.706763 21439 log.cc:824] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:46.707569 21439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347: No bootstrap required, opened a new log
Sep 26 04:11:46 dist-test-slave-xf24 krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:11:46.710021 21439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02839c286a814855906131bce6522347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260926 04:11:46.710198 21439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:46.710225 21439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02839c286a814855906131bce6522347, State: Initialized, Role: FOLLOWER
I20260926 04:11:46.710383 21439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02839c286a814855906131bce6522347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260926 04:11:46.710484 21439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:46.710515 21439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:46.710562 21439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:46.711277 21439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02839c286a814855906131bce6522347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260926 04:11:46.711371 21439 leader_election.cc:302] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02839c286a814855906131bce6522347; no voters:
I20260926 04:11:46.711620 21439 leader_election.cc:288] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:46.711649 21444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:46.711759 21444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [term 1 LEADER]: Becoming Leader. State: Replica: 02839c286a814855906131bce6522347, State: Running, Role: LEADER
I20260926 04:11:46.711920 21439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:46.711897 21444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02839c286a814855906131bce6522347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } }
I20260926 04:11:46.712415 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02839c286a814855906131bce6522347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02839c286a814855906131bce6522347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } } }
I20260926 04:11:46.712674 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:46.712589 21445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02839c286a814855906131bce6522347. Latest consensus state: current_term: 1 leader_uuid: "02839c286a814855906131bce6522347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02839c286a814855906131bce6522347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38101 } } }
I20260926 04:11:46.713096 21445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:46.713701 21453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:46.714272 21453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:46.714469 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:46.702459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50749 (local address 127.0.0.1:38101)
0926 04:11:46.702673 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:46.702687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:46.702713 (+ 26us) server_negotiation.cc:407] Connection header received
0926 04:11:46.703434 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:46.703442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:46.703500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:46.703677 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:46.704514 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:46.705463 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:46.706595 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:46.707182 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:46.707328 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:46.710081 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:46.710085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:46.710086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:46.710097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:46.712226 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:46.712675 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:46.712678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:46.712678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:46.712797 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:46.713027 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:46.713027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:46.713028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:46.713337 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:46.714000 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:46.714081 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:11:46.714222 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260926 04:11:46.715513 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:46.716048 21453 catalog_manager.cc:1347] Generated new cluster ID: cfaabf4f52e34fdbacc2d36a60e1a534
I20260926 04:11:46.716123 21453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:46.723110 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:46.716873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42735 (local address 127.0.0.1:38101)
0926 04:11:46.716917 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:46.716919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:46.717030 (+ 111us) server_negotiation.cc:407] Connection header received
0926 04:11:46.717107 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:46.717108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:46.717117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:46.717166 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:46.717675 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:46.718156 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:46.718896 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:46.719130 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:46.719220 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:46.720521 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:46.720522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:46.720522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:46.720529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:46.721931 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:46.722306 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:46.722307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:46.722307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:46.722363 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:46.722572 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:46.722573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:46.722573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:46.722703 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:46.722799 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:46.722922 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 04:11:46.723001 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:11:46.743662 21453 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:46.744316 21453 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:46.756817 21453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02839c286a814855906131bce6522347: Generated new TSK 0
I20260926 04:11:46.757058 21453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:46.764504 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21371
2026-09-26T04:11:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:46 dist-test-slave-xf24 krb5kdc[21470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:46 dist-test-slave-xf24 krb5kdc[21470](info): set up 2 sockets
Sep 26 04:11:46 dist-test-slave-xf24 krb5kdc[21470](info): commencing operation
krb5kdc: starting...
W20260926 04:11:47.844403 855 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:47 dist-test-slave-xf24 krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:47.968160 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:48.051237 21483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:48.051455 21483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:48.051478 21483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:48.052721 21483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:48.052771 21483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:48.052786 21483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:48.052799 21483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:48.052811 21483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:48.054286 21483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:48.054492 21483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:48.054666 21483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:11:48.059836 21483 server_base.cc:1034] running on GCE node
W20260926 04:11:48.059814 21489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:48.059814 21491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:48.059808 21488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:48.060309 21483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:48.060585 21483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:48.061735 21483 hybrid_clock.cc:648] HybridClock initialized: now 1790395908061720 us; error 34 us; skew 500 ppm
Sep 26 04:11:48 dist-test-slave-xf24 krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:48.068305 21483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:48.068854 21483 webserver.cc:460] Webserver started at http://127.0.0.1:42985/ using document root <none> and password file <none>
I20260926 04:11:48.069110 21483 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:48.069202 21483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:48.069358 21483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:48.070470 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb"
format_stamp: "Formatted at 2026-09-26 04:11:48 on dist-test-slave-xf24"
server_key: "ceb3729fd5a519c94d193aec089b4fe8"
server_key_iv: "02b56499bb86db8b0603f68d1cef8326"
server_key_version: "encryptionkey@0"
I20260926 04:11:48.070837 21483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb"
format_stamp: "Formatted at 2026-09-26 04:11:48 on dist-test-slave-xf24"
server_key: "ceb3729fd5a519c94d193aec089b4fe8"
server_key_iv: "02b56499bb86db8b0603f68d1cef8326"
server_key_version: "encryptionkey@0"
I20260926 04:11:48.072324 21483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:48.073180 21498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:48.073395 21483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:48.073498 21483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb"
format_stamp: "Formatted at 2026-09-26 04:11:48 on dist-test-slave-xf24"
server_key: "ceb3729fd5a519c94d193aec089b4fe8"
server_key_iv: "02b56499bb86db8b0603f68d1cef8326"
server_key_version: "encryptionkey@0"
I20260926 04:11:48.073635 21483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:48.106932 21483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:48.109380 21483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:48.109550 21483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:48.113718 21483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40797
I20260926 04:11:48.113757 21550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40797 every 8 connection(s)
I20260926 04:11:48.114104 21483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:48.114961 21551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:48.117389 21551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb: Bootstrap starting.
I20260926 04:11:48.117960 21551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:48.118247 21551 log.cc:824] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:48.119123 21551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb: No bootstrap required, opened a new log
I20260926 04:11:48.120817 21551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260926 04:11:48.120985 21551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:48.121044 21551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e0f8a1c33f4daa94cd6ee9e43f8cdb, State: Initialized, Role: FOLLOWER
I20260926 04:11:48.121224 21551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260926 04:11:48.121321 21551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:48.121366 21551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:48.121418 21551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:48.121953 21551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260926 04:11:48.122066 21551 leader_election.cc:302] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e0f8a1c33f4daa94cd6ee9e43f8cdb; no voters:
I20260926 04:11:48.122242 21551 leader_election.cc:288] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:48.122275 21554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:48.122473 21554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [term 1 LEADER]: Becoming Leader. State: Replica: 79e0f8a1c33f4daa94cd6ee9e43f8cdb, State: Running, Role: LEADER
I20260926 04:11:48.122522 21551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:48.122663 21554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } }
I20260926 04:11:48.123044 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e0f8a1c33f4daa94cd6ee9e43f8cdb. Latest consensus state: current_term: 1 leader_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } } }
I20260926 04:11:48.123142 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:48.123196 21555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e0f8a1c33f4daa94cd6ee9e43f8cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40797 } } }
I20260926 04:11:48.123255 21555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:48.123411 21562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:48.123668 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21483
I20260926 04:11:48.123795 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:48.123948 21562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:48.124006 855 external_mini_cluster.cc:1428] Setting key e49958b5ff8f33e3673310c622b165c2
I20260926 04:11:48.128545 21562 catalog_manager.cc:1347] Generated new cluster ID: f3a3b4d398fa4f909134a72893c8c25b
I20260926 04:11:48.128625 21562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:48.129263 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:48.124694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49587 (local address 127.0.0.1:40797)
0926 04:11:48.124890 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:48.124899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:48.124928 (+ 29us) server_negotiation.cc:407] Connection header received
0926 04:11:48.125062 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:48.125069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:48.125132 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:48.125273 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:48.125476 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:48.126410 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:48.127202 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:48.127625 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:48.127758 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:48.128542 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:48.128546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:48.128546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:48.128762 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:48.128806 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:48.128817 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:48.128894 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:48.128980 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 04:11:48.129046 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260926 04:11:48.130124 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:48.134349 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:48.131458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46223 (local address 127.0.0.1:40797)
0926 04:11:48.131494 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:48.131496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:48.131566 (+ 70us) server_negotiation.cc:407] Connection header received
0926 04:11:48.131626 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:48.131626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:48.131637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:48.131692 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:48.131795 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:48.132330 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:48.133023 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:48.133230 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:48.133299 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:48.134004 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:48.134006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:48.134006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:48.134011 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:48.134036 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:48.134045 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:48.134111 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:48.134209 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 04:11:48.134235 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 04:11:48.138051 21562 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:48.138631 21562 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:48.144584 21562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e0f8a1c33f4daa94cd6ee9e43f8cdb: Generated new TSK 0
I20260926 04:11:48.144829 21562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:48.151684 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21483
2026-09-26T04:11:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:48 dist-test-slave-xf24 krb5kdc[21582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:48 dist-test-slave-xf24 krb5kdc[21582](info): set up 2 sockets
Sep 26 04:11:48 dist-test-slave-xf24 krb5kdc[21582](info): commencing operation
krb5kdc: starting...
W20260926 04:11:49.212275 855 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:49 dist-test-slave-xf24 krb5kdc[21582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:49.339562 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:49.426863 21595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:49.427131 21595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:49.427173 21595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:49.428849 21595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:49.428932 21595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:49.428957 21595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:49.428987 21595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:49.429018 21595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:49.430646 21595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:49.430962 21595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:49.431205 21595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:49.436501 21601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:49.436539 21595 server_base.cc:1034] running on GCE node
W20260926 04:11:49.436525 21600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:49.436520 21603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:49.436990 21595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:49.437283 21595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:49.438447 21595 hybrid_clock.cc:648] HybridClock initialized: now 1790395909438443 us; error 36 us; skew 500 ppm
Sep 26 04:11:49 dist-test-slave-xf24 krb5kdc[21582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:49.441020 21595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:49.441468 21595 webserver.cc:460] Webserver started at http://127.0.0.1:40119/ using document root <none> and password file <none>
I20260926 04:11:49.441650 21595 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:49.441699 21595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:49.441788 21595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:49.442863 21595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "93cf49aceb7e44aba623ede05bf70513"
format_stamp: "Formatted at 2026-09-26 04:11:49 on dist-test-slave-xf24"
server_key: "e20b8f0b740ebbe67153a3e74859ab10"
server_key_iv: "1c354b34bc9a5f756e8c687325a64a8a"
server_key_version: "encryptionkey@0"
I20260926 04:11:49.443208 21595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "93cf49aceb7e44aba623ede05bf70513"
format_stamp: "Formatted at 2026-09-26 04:11:49 on dist-test-slave-xf24"
server_key: "e20b8f0b740ebbe67153a3e74859ab10"
server_key_iv: "1c354b34bc9a5f756e8c687325a64a8a"
server_key_version: "encryptionkey@0"
I20260926 04:11:49.444633 21595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:11:49.445377 21610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:49.445653 21595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:49.445753 21595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "93cf49aceb7e44aba623ede05bf70513"
format_stamp: "Formatted at 2026-09-26 04:11:49 on dist-test-slave-xf24"
server_key: "e20b8f0b740ebbe67153a3e74859ab10"
server_key_iv: "1c354b34bc9a5f756e8c687325a64a8a"
server_key_version: "encryptionkey@0"
I20260926 04:11:49.445811 21595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:49.449138 21595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:49.451414 21595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:49.451525 21595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:49.455618 21595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40509
I20260926 04:11:49.455643 21662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40509 every 8 connection(s)
I20260926 04:11:49.455992 21595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:49.457013 21663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:49.459275 21663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513: Bootstrap starting.
I20260926 04:11:49.459887 21663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:49.460242 21663 log.cc:824] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:49.460983 21663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513: No bootstrap required, opened a new log
I20260926 04:11:49.462648 21663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf49aceb7e44aba623ede05bf70513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260926 04:11:49.462823 21663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:49.462875 21663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cf49aceb7e44aba623ede05bf70513, State: Initialized, Role: FOLLOWER
I20260926 04:11:49.463059 21663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf49aceb7e44aba623ede05bf70513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260926 04:11:49.463155 21663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:49.463200 21663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:49.463249 21663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:49.463814 21663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf49aceb7e44aba623ede05bf70513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260926 04:11:49.463932 21663 leader_election.cc:302] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93cf49aceb7e44aba623ede05bf70513; no voters:
I20260926 04:11:49.464182 21663 leader_election.cc:288] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:49.464238 21666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:49.464433 21666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [term 1 LEADER]: Becoming Leader. State: Replica: 93cf49aceb7e44aba623ede05bf70513, State: Running, Role: LEADER
I20260926 04:11:49.464447 21663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:49.464591 21666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf49aceb7e44aba623ede05bf70513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } }
I20260926 04:11:49.464964 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21595
I20260926 04:11:49.464978 21668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cf49aceb7e44aba623ede05bf70513. Latest consensus state: current_term: 1 leader_uuid: "93cf49aceb7e44aba623ede05bf70513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf49aceb7e44aba623ede05bf70513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } } }
I20260926 04:11:49.465049 21668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:49.465080 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:49.465128 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93cf49aceb7e44aba623ede05bf70513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cf49aceb7e44aba623ede05bf70513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40509 } } }
I20260926 04:11:49.465216 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:49.465274 855 external_mini_cluster.cc:1428] Setting key c821a5215e2491cc5b7989cd6273813a
I20260926 04:11:49.465281 21673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:49.465791 21673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:49.467525 21673 catalog_manager.cc:1347] Generated new cluster ID: e9cd4f9a812240829bf3a002cbadcc3e
I20260926 04:11:49.467584 21673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:49.468210 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:49.465925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35347 (local address 127.0.0.1:40509)
0926 04:11:49.466488 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:49.466498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:49.466540 (+ 42us) server_negotiation.cc:407] Connection header received
0926 04:11:49.466729 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:49.466738 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:49.466769 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:49.466926 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:49.467621 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:49.467625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:49.467625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:49.467629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:49.467673 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:49.467687 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:49.467742 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:49.467963 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 04:11:49.468002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":70,"threads_started":1}
I20260926 04:11:49.469076 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:49.471704 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:49.470599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46917 (local address 127.0.0.1:40509)
0926 04:11:49.470627 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:49.470629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:49.470745 (+ 116us) server_negotiation.cc:407] Connection header received
0926 04:11:49.470813 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:49.470814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:49.470819 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:49.470874 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:49.471490 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:49.471492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:49.471492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:49.471495 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:49.471509 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:49.471513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:49.471565 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:49.471631 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 04:11:49.471643 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 04:11:49.482415 21673 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:49.482611 21673 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:49.486390 21673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93cf49aceb7e44aba623ede05bf70513: Generated new TSK 0
I20260926 04:11:49.486642 21673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:49.492251 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21595
2026-09-26T04:11:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:49 dist-test-slave-xf24 krb5kdc[21694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:49 dist-test-slave-xf24 krb5kdc[21694](info): set up 2 sockets
Sep 26 04:11:49 dist-test-slave-xf24 krb5kdc[21694](info): commencing operation
krb5kdc: starting...
W20260926 04:11:50.528288 855 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:50 dist-test-slave-xf24 krb5kdc[21694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:50.651353 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:50.739363 21707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:50.739657 21707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:50.739712 21707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:50.741014 21707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:50.741113 21707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:50.741137 21707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:50.741159 21707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:50.741181 21707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:50.742893 21707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:50.743247 21707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:50.743513 21707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:11:50.750182 21707 server_base.cc:1034] running on GCE node
W20260926 04:11:50.750070 21715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:50.750070 21712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:50.750094 21713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:50.750775 21707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:50.751082 21707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:50.752247 21707 hybrid_clock.cc:648] HybridClock initialized: now 1790395910752232 us; error 43 us; skew 500 ppm
Sep 26 04:11:50 dist-test-slave-xf24 krb5kdc[21694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:50.758847 21707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:50.759411 21707 webserver.cc:460] Webserver started at http://127.0.0.1:40999/ using document root <none> and password file <none>
I20260926 04:11:50.759680 21707 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:50.759756 21707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:50.759920 21707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:50.761310 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "7f66ab64ca0a4a0990ec32cd8290e046"
format_stamp: "Formatted at 2026-09-26 04:11:50 on dist-test-slave-xf24"
server_key: "9dbb2762f719f890b1e966edca7a8943"
server_key_iv: "79ed0d3dd3b999abd98bca891966ddf7"
server_key_version: "encryptionkey@0"
I20260926 04:11:50.761741 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "7f66ab64ca0a4a0990ec32cd8290e046"
format_stamp: "Formatted at 2026-09-26 04:11:50 on dist-test-slave-xf24"
server_key: "9dbb2762f719f890b1e966edca7a8943"
server_key_iv: "79ed0d3dd3b999abd98bca891966ddf7"
server_key_version: "encryptionkey@0"
I20260926 04:11:50.763296 21707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:50.764269 21722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:50.764477 21707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:50.764592 21707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "7f66ab64ca0a4a0990ec32cd8290e046"
format_stamp: "Formatted at 2026-09-26 04:11:50 on dist-test-slave-xf24"
server_key: "9dbb2762f719f890b1e966edca7a8943"
server_key_iv: "79ed0d3dd3b999abd98bca891966ddf7"
server_key_version: "encryptionkey@0"
I20260926 04:11:50.764686 21707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:50.781809 21707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:50.785291 21707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:50.785547 21707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:50.790583 21707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33479
I20260926 04:11:50.790701 21774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33479 every 8 connection(s)
I20260926 04:11:50.791119 21707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:50.792037 21775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:50.794430 21775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046: Bootstrap starting.
I20260926 04:11:50.795006 21775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:50.795260 21775 log.cc:824] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:50.796015 21775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046: No bootstrap required, opened a new log
I20260926 04:11:50.797524 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21707
I20260926 04:11:50.797652 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:50.797854 855 external_mini_cluster.cc:1428] Setting key b7910d48dd33d2ba9bc34cc7e050a369
I20260926 04:11:50.798061 21775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33479 } }
I20260926 04:11:50.798300 21775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:50.798349 21775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f66ab64ca0a4a0990ec32cd8290e046, State: Initialized, Role: FOLLOWER
I20260926 04:11:50.798533 21775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33479 } }
I20260926 04:11:50.798647 21775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:50.798696 21775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:50.798751 21775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:50.799894 21775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33479 } }
I20260926 04:11:50.800024 21775 leader_election.cc:302] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f66ab64ca0a4a0990ec32cd8290e046; no voters:
I20260926 04:11:50.800343 21775 leader_election.cc:288] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:50.800366 21780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:50.800518 21780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [term 1 LEADER]: Becoming Leader. State: Replica: 7f66ab64ca0a4a0990ec32cd8290e046, State: Running, Role: LEADER
I20260926 04:11:50.800614 21780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33479 } }
I20260926 04:11:50.800673 21775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:50.801040 21781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33479 } } }
I20260926 04:11:50.801191 21781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:50.801072 21782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f66ab64ca0a4a0990ec32cd8290e046. Latest consensus state: current_term: 1 leader_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f66ab64ca0a4a0990ec32cd8290e046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33479 } } }
I20260926 04:11:50.801419 21782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:50.801437 21784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:50.802038 21784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 04:11:50 dist-test-slave-xf24 krb5kdc[21694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:11:50.810416 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:50.798452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:32771 (local address 127.0.0.1:33479)
0926 04:11:50.798886 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:50.798896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:50.798932 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:11:50.799061 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:50.799069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:50.799155 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:50.799343 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:50.800138 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:50.801612 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:50.802818 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:50.803375 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:50.803497 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:50.806511 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:50.806516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:50.806516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:50.806530 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:50.808551 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:50.809016 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:50.809020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:50.809020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:50.809139 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:50.809392 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:50.809393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:50.809394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:50.809699 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:50.809908 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:50.810063 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 04:11:50.810183 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":320,"threads_started":1}
I20260926 04:11:50.810437 21784 catalog_manager.cc:1347] Generated new cluster ID: 47860ded23e244e1909c3eeac7d4c078
I20260926 04:11:50.810942 21784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:50.811714 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:50.821496 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:50.813453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:47693 (local address 127.0.0.1:33479)
0926 04:11:50.813498 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:50.813500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:50.813532 (+ 32us) server_negotiation.cc:407] Connection header received
0926 04:11:50.813555 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:50.813556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:50.813566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:50.813620 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:50.814276 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:50.815054 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:50.816230 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:50.816587 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:50.816665 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:50.818225 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:50.818227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:50.818227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:50.818235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:50.820117 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:50.820578 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:50.820580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:50.820580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:50.820651 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:50.820900 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:50.820901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:50.820901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:50.821050 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:50.821170 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:50.821320 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 04:11:50.821390 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:11:50.849411 21784 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:50.850193 21784 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:50.858875 21784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f66ab64ca0a4a0990ec32cd8290e046: Generated new TSK 0
I20260926 04:11:50.859145 21784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:50.863780 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21707
W20260926 04:11:50.868294 21692 connection.cc:535] client connection to 127.0.0.1:33479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33479): Connection reset by peer (error 104)
2026-09-26T04:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:50 dist-test-slave-xf24 krb5kdc[21806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:50 dist-test-slave-xf24 krb5kdc[21806](info): set up 2 sockets
Sep 26 04:11:50 dist-test-slave-xf24 krb5kdc[21806](info): commencing operation
krb5kdc: starting...
W20260926 04:11:51.908334 855 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:51 dist-test-slave-xf24 krb5kdc[21806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:52.032704 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:52.114596 21819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:52.114823 21819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:52.114848 21819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:52.115949 21819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:52.115985 21819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:52.115999 21819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:52.116012 21819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:52.116024 21819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:52.117522 21819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:52.117719 21819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:52.117899 21819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:52.122743 21824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:52.122725 21827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:52.122830 21819 server_base.cc:1034] running on GCE node
W20260926 04:11:52.122718 21825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:52.123157 21819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:52.123442 21819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:52.124634 21819 hybrid_clock.cc:648] HybridClock initialized: now 1790395912124611 us; error 49 us; skew 500 ppm
Sep 26 04:11:52 dist-test-slave-xf24 krb5kdc[21806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:52.127313 21819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:52.127735 21819 webserver.cc:460] Webserver started at http://127.0.0.1:46653/ using document root <none> and password file <none>
I20260926 04:11:52.127979 21819 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:52.128046 21819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:52.128202 21819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:52.129256 21819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "c6eca5e2ff344043aabd76d86127d2a7"
format_stamp: "Formatted at 2026-09-26 04:11:52 on dist-test-slave-xf24"
server_key: "3e82546cdc9c0c1c2085eac13deb4355"
server_key_iv: "2519defa19e4e45600c4ade3e6c387d6"
server_key_version: "encryptionkey@0"
I20260926 04:11:52.129611 21819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "c6eca5e2ff344043aabd76d86127d2a7"
format_stamp: "Formatted at 2026-09-26 04:11:52 on dist-test-slave-xf24"
server_key: "3e82546cdc9c0c1c2085eac13deb4355"
server_key_iv: "2519defa19e4e45600c4ade3e6c387d6"
server_key_version: "encryptionkey@0"
I20260926 04:11:52.130919 21819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:52.131609 21834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:52.131779 21819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:11:52.131897 21819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "c6eca5e2ff344043aabd76d86127d2a7"
format_stamp: "Formatted at 2026-09-26 04:11:52 on dist-test-slave-xf24"
server_key: "3e82546cdc9c0c1c2085eac13deb4355"
server_key_iv: "2519defa19e4e45600c4ade3e6c387d6"
server_key_version: "encryptionkey@0"
I20260926 04:11:52.131973 21819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:52.147249 21819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:52.149682 21819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:52.149878 21819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:52.153788 21819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38393
I20260926 04:11:52.153846 21886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38393 every 8 connection(s)
I20260926 04:11:52.154201 21819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:52.155061 21887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:52.157449 21887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7: Bootstrap starting.
I20260926 04:11:52.157727 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21819
I20260926 04:11:52.157856 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:52.158049 855 external_mini_cluster.cc:1428] Setting key 14a87e46f6b626360aafc0eb17c1697f
I20260926 04:11:52.158138 21887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:52.158416 21887 log.cc:824] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:52.159129 21887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7: No bootstrap required, opened a new log
I20260926 04:11:52.161404 21887 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6eca5e2ff344043aabd76d86127d2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260926 04:11:52.161600 21887 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:52.161631 21887 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6eca5e2ff344043aabd76d86127d2a7, State: Initialized, Role: FOLLOWER
I20260926 04:11:52.161754 21887 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6eca5e2ff344043aabd76d86127d2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260926 04:11:52.161845 21887 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:52.161886 21887 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:52.161933 21887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:52.162583 21887 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6eca5e2ff344043aabd76d86127d2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260926 04:11:52.162714 21887 leader_election.cc:302] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6eca5e2ff344043aabd76d86127d2a7; no voters:
I20260926 04:11:52.162927 21887 leader_election.cc:288] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:52.162984 21892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:52.163205 21887 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:52.163223 21892 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [term 1 LEADER]: Becoming Leader. State: Replica: c6eca5e2ff344043aabd76d86127d2a7, State: Running, Role: LEADER
I20260926 04:11:52.163407 21892 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6eca5e2ff344043aabd76d86127d2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } }
I20260926 04:11:52.164171 21894 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6eca5e2ff344043aabd76d86127d2a7. Latest consensus state: current_term: 1 leader_uuid: "c6eca5e2ff344043aabd76d86127d2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6eca5e2ff344043aabd76d86127d2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } } }
I20260926 04:11:52.164326 21894 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:52.164340 21893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6eca5e2ff344043aabd76d86127d2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6eca5e2ff344043aabd76d86127d2a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38393 } } }
I20260926 04:11:52.164499 21893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:52.164737 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:52.158592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:33439 (local address 127.0.0.1:38393)
0926 04:11:52.158776 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:52.158784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:52.158817 (+ 33us) server_negotiation.cc:407] Connection header received
0926 04:11:52.159305 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:52.159312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:52.159362 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:52.159483 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:52.159738 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:52.160740 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:52.161902 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:52.162405 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:52.162499 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:52.163776 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:52.163781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:52.163781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:52.163997 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:52.164037 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:52.164055 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:52.164174 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:52.164453 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 04:11:52.164485 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260926 04:11:52.164992 21904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:52.165654 21904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:52.165709 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:52.167419 21904 catalog_manager.cc:1347] Generated new cluster ID: aed5a67b026a423597d62ec972347294
I20260926 04:11:52.167485 21904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:52.170166 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:52.167270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:37585 (local address 127.0.0.1:38393)
0926 04:11:52.167313 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:52.167315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:52.167401 (+ 86us) server_negotiation.cc:407] Connection header received
0926 04:11:52.167481 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:52.167482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:52.167493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:52.167550 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:52.167695 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:52.168250 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:52.168959 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:52.169194 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:52.169265 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:52.169942 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:52.169943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:52.169944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:52.169948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:52.169963 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:52.169967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:52.170020 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:52.170079 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 04:11:52.170089 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:11:52.193023 21904 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:52.193775 21904 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:52.205107 21904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6eca5e2ff344043aabd76d86127d2a7: Generated new TSK 0
I20260926 04:11:52.205322 21904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:52.215617 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21819
W20260926 04:11:52.220098 21804 connection.cc:535] client connection to 127.0.0.1:38393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38393): Connection reset by peer (error 104)
2026-09-26T04:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:52 dist-test-slave-xf24 krb5kdc[21918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:52 dist-test-slave-xf24 krb5kdc[21918](info): set up 2 sockets
Sep 26 04:11:52 dist-test-slave-xf24 krb5kdc[21918](info): commencing operation
krb5kdc: starting...
W20260926 04:11:53.284358 855 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:53 dist-test-slave-xf24 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:53.410956 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:53.491838 21931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:53.492040 21931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:53.492111 21931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:53.493278 21931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:53.493319 21931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:53.493333 21931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:53.493346 21931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:53.493359 21931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:53.494838 21931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:53.495074 21931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:53.495288 21931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:53.500299 21937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:53.500310 21939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:53.500313 21936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:53.500620 21931 server_base.cc:1034] running on GCE node
I20260926 04:11:53.500866 21931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:53.501147 21931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:53.502341 21931 hybrid_clock.cc:648] HybridClock initialized: now 1790395913502322 us; error 38 us; skew 500 ppm
Sep 26 04:11:53 dist-test-slave-xf24 krb5kdc[21918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:53.509573 21931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:53.510126 21931 webserver.cc:460] Webserver started at http://127.0.0.1:37097/ using document root <none> and password file <none>
I20260926 04:11:53.510390 21931 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:53.510457 21931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:53.510620 21931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:53.511729 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "3dee2845de3347d59328a9fa18139a6d"
format_stamp: "Formatted at 2026-09-26 04:11:53 on dist-test-slave-xf24"
server_key: "5c46fe33bb53ac28c3b8db3409936818"
server_key_iv: "d7e550a973c83bd32d6578da9885bc73"
server_key_version: "encryptionkey@0"
I20260926 04:11:53.512168 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "3dee2845de3347d59328a9fa18139a6d"
format_stamp: "Formatted at 2026-09-26 04:11:53 on dist-test-slave-xf24"
server_key: "5c46fe33bb53ac28c3b8db3409936818"
server_key_iv: "d7e550a973c83bd32d6578da9885bc73"
server_key_version: "encryptionkey@0"
I20260926 04:11:53.513581 21931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:53.514439 21946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:53.514722 21931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:53.514848 21931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "3dee2845de3347d59328a9fa18139a6d"
format_stamp: "Formatted at 2026-09-26 04:11:53 on dist-test-slave-xf24"
server_key: "5c46fe33bb53ac28c3b8db3409936818"
server_key_iv: "d7e550a973c83bd32d6578da9885bc73"
server_key_version: "encryptionkey@0"
I20260926 04:11:53.514925 21931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:53.519871 21931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:53.522794 21931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:53.522958 21931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:53.527441 21931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46485
I20260926 04:11:53.527493 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46485 every 8 connection(s)
I20260926 04:11:53.527807 21931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:53.528931 21999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:53.531224 21999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d: Bootstrap starting.
I20260926 04:11:53.531759 21999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:53.531993 21999 log.cc:824] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:53.532779 21999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d: No bootstrap required, opened a new log
I20260926 04:11:53.534580 21999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dee2845de3347d59328a9fa18139a6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260926 04:11:53.534736 21999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:53.534761 21999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dee2845de3347d59328a9fa18139a6d, State: Initialized, Role: FOLLOWER
I20260926 04:11:53.534926 21999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dee2845de3347d59328a9fa18139a6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260926 04:11:53.535018 21999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:53.535044 21999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:53.535076 21999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:53.535707 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 21931
I20260926 04:11:53.535743 21999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dee2845de3347d59328a9fa18139a6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260926 04:11:53.535887 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:53.536001 21999 leader_election.cc:302] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dee2845de3347d59328a9fa18139a6d; no voters:
I20260926 04:11:53.536180 855 external_mini_cluster.cc:1428] Setting key 766cd41991798602e992f11e23b94232
I20260926 04:11:53.536280 21999 leader_election.cc:288] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:53.536341 22002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:53.536511 22002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [term 1 LEADER]: Becoming Leader. State: Replica: 3dee2845de3347d59328a9fa18139a6d, State: Running, Role: LEADER
I20260926 04:11:53.536567 21999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:53.536610 22002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dee2845de3347d59328a9fa18139a6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260926 04:11:53.537039 22004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dee2845de3347d59328a9fa18139a6d. Latest consensus state: current_term: 1 leader_uuid: "3dee2845de3347d59328a9fa18139a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dee2845de3347d59328a9fa18139a6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } } }
I20260926 04:11:53.537139 22004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:53.537050 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dee2845de3347d59328a9fa18139a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dee2845de3347d59328a9fa18139a6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } } }
I20260926 04:11:53.537232 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:53.537349 22008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:53.537976 22008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:53.539309 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:53.536818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:37417 (local address 127.0.0.1:46485)
0926 04:11:53.537257 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:53.537267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:53.537304 (+ 37us) server_negotiation.cc:407] Connection header received
0926 04:11:53.537446 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:53.537453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:53.537491 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:53.537647 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:53.538833 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:53.538837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:53.538837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:53.538842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:53.538890 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:53.538904 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:53.538983 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:53.539077 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 04:11:53.539124 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":68,"threads_started":1}
I20260926 04:11:53.540079 22008 catalog_manager.cc:1347] Generated new cluster ID: ad5b95cebdc84a3e91733b87f4ecd688
I20260926 04:11:53.540133 22008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:53.540359 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:53.543143 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:53.541956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:51457 (local address 127.0.0.1:46485)
0926 04:11:53.541994 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:53.541996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:53.542068 (+ 72us) server_negotiation.cc:407] Connection header received
0926 04:11:53.542131 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:53.542132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:53.542148 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:53.542257 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:53.542907 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:53.542908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:53.542909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:53.542913 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:53.542931 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:53.542936 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:53.542993 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:53.543055 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 04:11:53.543070 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:11:53.547511 22008 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:53.547708 22008 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:53.552748 22008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dee2845de3347d59328a9fa18139a6d: Generated new TSK 0
I20260926 04:11:53.552953 22008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:53.555414 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 21931
2026-09-26T04:11:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:53 dist-test-slave-xf24 krb5kdc[22030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:53 dist-test-slave-xf24 krb5kdc[22030](info): set up 2 sockets
Sep 26 04:11:53 dist-test-slave-xf24 krb5kdc[22030](info): commencing operation
krb5kdc: starting...
W20260926 04:11:54.612558 855 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:54 dist-test-slave-xf24 krb5kdc[22030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:54.739173 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:54.820344 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:54.820564 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:54.820587 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:54.821698 22043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:54.821736 22043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:54.821751 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:54.821763 22043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:54.821775 22043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:54.823257 22043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:54.823515 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:54.823729 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:54.829125 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:54.829125 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:54.829466 22043 server_base.cc:1034] running on GCE node
W20260926 04:11:54.829149 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:54.829773 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:54.830062 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:54.831251 22043 hybrid_clock.cc:648] HybridClock initialized: now 1790395914831212 us; error 55 us; skew 500 ppm
Sep 26 04:11:54 dist-test-slave-xf24 krb5kdc[22030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:54.834022 22043 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:54.834455 22043 webserver.cc:460] Webserver started at http://127.0.0.1:45117/ using document root <none> and password file <none>
I20260926 04:11:54.834669 22043 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:54.834715 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:54.834805 22043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:54.835837 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "0ec074a98012456388068f364cd8bcb6"
format_stamp: "Formatted at 2026-09-26 04:11:54 on dist-test-slave-xf24"
server_key: "ad0700a3b5159f16d02a10a7a91a3a7d"
server_key_iv: "32563d77f1b10040b9cfd19a2d72b912"
server_key_version: "encryptionkey@0"
I20260926 04:11:54.836212 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "0ec074a98012456388068f364cd8bcb6"
format_stamp: "Formatted at 2026-09-26 04:11:54 on dist-test-slave-xf24"
server_key: "ad0700a3b5159f16d02a10a7a91a3a7d"
server_key_iv: "32563d77f1b10040b9cfd19a2d72b912"
server_key_version: "encryptionkey@0"
I20260926 04:11:54.837627 22043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:54.838440 22058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:54.838627 22043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:54.838729 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "0ec074a98012456388068f364cd8bcb6"
format_stamp: "Formatted at 2026-09-26 04:11:54 on dist-test-slave-xf24"
server_key: "ad0700a3b5159f16d02a10a7a91a3a7d"
server_key_iv: "32563d77f1b10040b9cfd19a2d72b912"
server_key_version: "encryptionkey@0"
I20260926 04:11:54.838780 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:54.861385 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:54.863772 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:54.863917 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:54.867782 22043 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33481
I20260926 04:11:54.867856 22110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33481 every 8 connection(s)
I20260926 04:11:54.868251 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:54.869174 22111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:54.871455 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6: Bootstrap starting.
I20260926 04:11:54.872094 22111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:54.872386 22111 log.cc:824] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:54.873124 22111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6: No bootstrap required, opened a new log
I20260926 04:11:54.874302 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22043
I20260926 04:11:54.874400 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:54.874603 855 external_mini_cluster.cc:1428] Setting key 872d2a899f3fb53cfa003a8d83301057
I20260926 04:11:54.875077 22111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec074a98012456388068f364cd8bcb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260926 04:11:54.875269 22111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:54.875337 22111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec074a98012456388068f364cd8bcb6, State: Initialized, Role: FOLLOWER
I20260926 04:11:54.875474 22111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec074a98012456388068f364cd8bcb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260926 04:11:54.875568 22111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:54.875610 22111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:54.875658 22111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:54.876878 22111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec074a98012456388068f364cd8bcb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260926 04:11:54.876995 22111 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec074a98012456388068f364cd8bcb6; no voters:
I20260926 04:11:54.877161 22111 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:54.877200 22116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:54.877385 22116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec074a98012456388068f364cd8bcb6, State: Running, Role: LEADER
I20260926 04:11:54.877415 22111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:54.877496 22116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec074a98012456388068f364cd8bcb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260926 04:11:54.877805 22116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec074a98012456388068f364cd8bcb6. Latest consensus state: current_term: 1 leader_uuid: "0ec074a98012456388068f364cd8bcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec074a98012456388068f364cd8bcb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260926 04:11:54.877813 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec074a98012456388068f364cd8bcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec074a98012456388068f364cd8bcb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260926 04:11:54.877897 22116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:54.877929 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6 [sys.catalog]: This master's current role is: LEADER
W20260926 04:11:54.879369 22132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:11:54.879467 22132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:11:54.879556 22122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:54.879715 22122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:54.881381 22122 catalog_manager.cc:1347] Generated new cluster ID: aeb09ec704ee4754ac692c4a6634213f
I20260926 04:11:54.881435 22122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:11:54 dist-test-slave-xf24 krb5kdc[22030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:11:54.885493 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:54.875155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:60469 (local address 127.0.0.1:33481)
0926 04:11:54.875430 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:54.875440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:54.875474 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:11:54.875594 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:54.875601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:54.875660 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:54.875792 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:54.876633 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:54.877930 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:54.878971 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:54.879434 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:54.879549 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:54.882195 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:54.882199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:54.882200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:54.882212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:54.883927 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:54.884374 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:54.884376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:54.884377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:54.884477 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:54.884708 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:54.884709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:54.884709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:54.884945 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:54.885129 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:54.885226 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 04:11:54.885340 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
I20260926 04:11:54.886626 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:54.890077 22122 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:54.890775 22122 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:54.894407 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:54.887963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:44743 (local address 127.0.0.1:33481)
0926 04:11:54.888003 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:54.888005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:54.888217 (+ 212us) server_negotiation.cc:407] Connection header received
0926 04:11:54.888292 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:54.888293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:54.888310 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:54.888352 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:54.888879 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:54.889389 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:54.890321 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:54.890541 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:54.890612 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:54.891839 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:54.891841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:54.891841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:54.891848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:54.893254 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:54.893610 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:54.893611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:54.893612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:54.893673 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:54.893887 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:54.893888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:54.893888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:54.893997 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:54.894117 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:54.894227 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 04:11:54.894289 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":20}
I20260926 04:11:54.897455 22122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec074a98012456388068f364cd8bcb6: Generated new TSK 0
I20260926 04:11:54.897796 22122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:54.902496 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22043
W20260926 04:11:54.906685 22028 connection.cc:535] client connection to 127.0.0.1:33481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33481): Connection reset by peer (error 104)
2026-09-26T04:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:54 dist-test-slave-xf24 krb5kdc[22142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:54 dist-test-slave-xf24 krb5kdc[22142](info): set up 2 sockets
Sep 26 04:11:54 dist-test-slave-xf24 krb5kdc[22142](info): commencing operation
krb5kdc: starting...
W20260926 04:11:55.944307 855 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:56 dist-test-slave-xf24 krb5kdc[22142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:56.067734 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:56.153681 22155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:56.153951 22155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:56.153992 22155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:56.155653 22155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:56.155766 22155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:56.155794 22155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:56.155828 22155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:56.155856 22155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:56.157604 22155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:56.157920 22155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:56.158160 22155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:56.163537 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:56.163542 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:56.163658 22155 server_base.cc:1034] running on GCE node
W20260926 04:11:56.163542 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:56.164243 22155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:56.164597 22155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:56.165791 22155 hybrid_clock.cc:648] HybridClock initialized: now 1790395916165767 us; error 37 us; skew 500 ppm
Sep 26 04:11:56 dist-test-slave-xf24 krb5kdc[22142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:56.168661 22155 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:56.169211 22155 webserver.cc:460] Webserver started at http://127.0.0.1:33791/ using document root <none> and password file <none>
I20260926 04:11:56.169479 22155 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:56.169549 22155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:56.169699 22155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:56.170816 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "39db1afd9d53451bb5e5d354687b5415"
format_stamp: "Formatted at 2026-09-26 04:11:56 on dist-test-slave-xf24"
server_key: "8feb2a31c3bf7339beff8d6df2fe0c0b"
server_key_iv: "ced2008edac413bb9d91cf9bc6e98a52"
server_key_version: "encryptionkey@0"
I20260926 04:11:56.171187 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "39db1afd9d53451bb5e5d354687b5415"
format_stamp: "Formatted at 2026-09-26 04:11:56 on dist-test-slave-xf24"
server_key: "8feb2a31c3bf7339beff8d6df2fe0c0b"
server_key_iv: "ced2008edac413bb9d91cf9bc6e98a52"
server_key_version: "encryptionkey@0"
I20260926 04:11:56.172745 22155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:11:56.173550 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:56.173727 22155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:11:56.173836 22155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "39db1afd9d53451bb5e5d354687b5415"
format_stamp: "Formatted at 2026-09-26 04:11:56 on dist-test-slave-xf24"
server_key: "8feb2a31c3bf7339beff8d6df2fe0c0b"
server_key_iv: "ced2008edac413bb9d91cf9bc6e98a52"
server_key_version: "encryptionkey@0"
I20260926 04:11:56.173918 22155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:56.190991 22155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:56.193408 22155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:56.193583 22155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:56.197886 22155 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43171
I20260926 04:11:56.197924 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43171 every 8 connection(s)
I20260926 04:11:56.198284 22155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:56.199306 22223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:56.201639 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415: Bootstrap starting.
I20260926 04:11:56.202178 22223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:56.202412 22223 log.cc:824] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:56.203227 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22155
I20260926 04:11:56.203361 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:56.203414 22223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415: No bootstrap required, opened a new log
I20260926 04:11:56.203562 855 external_mini_cluster.cc:1428] Setting key a5c1001be995591394d5a747d8d42621
I20260926 04:11:56.206067 22223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39db1afd9d53451bb5e5d354687b5415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260926 04:11:56.206200 22223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:56.206223 22223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39db1afd9d53451bb5e5d354687b5415, State: Initialized, Role: FOLLOWER
I20260926 04:11:56.206348 22223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39db1afd9d53451bb5e5d354687b5415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260926 04:11:56.206445 22223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:56.206488 22223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:56.206540 22223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:56.207078 22223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39db1afd9d53451bb5e5d354687b5415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260926 04:11:56.207195 22223 leader_election.cc:302] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39db1afd9d53451bb5e5d354687b5415; no voters:
I20260926 04:11:56.207402 22223 leader_election.cc:288] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:56.207423 22228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:56.207542 22228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [term 1 LEADER]: Becoming Leader. State: Replica: 39db1afd9d53451bb5e5d354687b5415, State: Running, Role: LEADER
I20260926 04:11:56.207674 22223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:56.207675 22228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39db1afd9d53451bb5e5d354687b5415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } }
I20260926 04:11:56.208128 22230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39db1afd9d53451bb5e5d354687b5415. Latest consensus state: current_term: 1 leader_uuid: "39db1afd9d53451bb5e5d354687b5415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39db1afd9d53451bb5e5d354687b5415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } } }
I20260926 04:11:56.208235 22230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:56.208461 22233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:56.208770 22229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39db1afd9d53451bb5e5d354687b5415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39db1afd9d53451bb5e5d354687b5415" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43171 } } }
I20260926 04:11:56.208855 22229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:56.208972 22233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:56.209620 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:56.204147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:49817 (local address 127.0.0.1:43171)
0926 04:11:56.204314 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:56.204330 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:56.204359 (+ 29us) server_negotiation.cc:407] Connection header received
0926 04:11:56.204802 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:56.204809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:56.204857 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:56.204968 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:56.205210 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:56.206305 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:56.207219 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:56.207870 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:56.207975 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:56.208778 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:56.208782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:56.208782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:56.208998 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:56.209041 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:56.209058 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:56.209148 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:56.209382 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 04:11:56.209411 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260926 04:11:56.210652 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:56.214883 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:56.211932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:53597 (local address 127.0.0.1:43171)
0926 04:11:56.211976 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:56.211978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:56.212086 (+ 108us) server_negotiation.cc:407] Connection header received
0926 04:11:56.212197 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:56.212198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:56.212208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:56.212260 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:56.212412 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:56.212886 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:56.213605 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:56.213849 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:56.213937 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:56.214643 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:56.214644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:56.214645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:56.214649 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:56.214663 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:56.214668 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:56.214722 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:56.214782 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 04:11:56.214791 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:11:56.215490 22233 catalog_manager.cc:1347] Generated new cluster ID: 0ca5394f3cf34c3380e5592775073735
I20260926 04:11:56.215552 22233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:56.237568 22233 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:56.238157 22233 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:56.253918 22233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39db1afd9d53451bb5e5d354687b5415: Generated new TSK 0
I20260926 04:11:56.254149 22233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:56.262609 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22155
W20260926 04:11:56.267179 22140 connection.cc:535] client connection to 127.0.0.1:43171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43171): Connection reset by peer (error 104)
2026-09-26T04:11:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:56 dist-test-slave-xf24 krb5kdc[22254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:56 dist-test-slave-xf24 krb5kdc[22254](info): set up 2 sockets
Sep 26 04:11:56 dist-test-slave-xf24 krb5kdc[22254](info): commencing operation
krb5kdc: starting...
W20260926 04:11:57.300424 855 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:57 dist-test-slave-xf24 krb5kdc[22254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:11:57.428138 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:57.511467 22267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:57.511694 22267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:57.511719 22267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:57.513063 22267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:57.513111 22267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:57.513126 22267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:57.513139 22267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:57.513152 22267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:57.514686 22267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:57.514962 22267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:57.515204 22267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:57.520311 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:57.520332 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:57.520299 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:57.520553 22267 server_base.cc:1034] running on GCE node
I20260926 04:11:57.520784 22267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:57.521039 22267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:57.522231 22267 hybrid_clock.cc:648] HybridClock initialized: now 1790395917522224 us; error 39 us; skew 500 ppm
Sep 26 04:11:57 dist-test-slave-xf24 krb5kdc[22254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:57.525323 22267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:11:57.525804 22267 webserver.cc:460] Webserver started at http://127.0.0.1:45659/ using document root <none> and password file <none>
I20260926 04:11:57.526093 22267 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:57.526141 22267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:57.526261 22267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:57.527355 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "88b47fa9f00943be916f05345364b4c4"
format_stamp: "Formatted at 2026-09-26 04:11:57 on dist-test-slave-xf24"
server_key: "fec6f9840365e08e8f9b22e1c2738747"
server_key_iv: "f69b22a0cc9a4e5126125f6e22baf330"
server_key_version: "encryptionkey@0"
I20260926 04:11:57.527720 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "88b47fa9f00943be916f05345364b4c4"
format_stamp: "Formatted at 2026-09-26 04:11:57 on dist-test-slave-xf24"
server_key: "fec6f9840365e08e8f9b22e1c2738747"
server_key_iv: "f69b22a0cc9a4e5126125f6e22baf330"
server_key_version: "encryptionkey@0"
I20260926 04:11:57.529111 22267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:11:57.530004 22282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:57.530198 22267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:57.530303 22267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "88b47fa9f00943be916f05345364b4c4"
format_stamp: "Formatted at 2026-09-26 04:11:57 on dist-test-slave-xf24"
server_key: "fec6f9840365e08e8f9b22e1c2738747"
server_key_iv: "f69b22a0cc9a4e5126125f6e22baf330"
server_key_version: "encryptionkey@0"
I20260926 04:11:57.530395 22267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:57.533815 22267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:57.536195 22267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:57.536350 22267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:57.540575 22267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260926 04:11:57.540612 22334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260926 04:11:57.540977 22267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:57.542017 22335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:57.543126 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22267
I20260926 04:11:57.543272 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:57.543483 855 external_mini_cluster.cc:1428] Setting key d4ecd3ae294fcaa4a5b108cbe859ad6d
I20260926 04:11:57.545096 22335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4: Bootstrap starting.
I20260926 04:11:57.545902 22335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:57.546300 22335 log.cc:824] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:57.546602 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:57.544124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:37957 (local address 127.0.0.1:35665)
0926 04:11:57.544341 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:57.544351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:57.544389 (+ 38us) server_negotiation.cc:407] Connection header received
0926 04:11:57.544968 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:57.544978 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:57.545023 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:57.545189 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:57.546018 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:57.546022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:57.546022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:57.546026 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:57.546080 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:57.546096 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:57.546192 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:57.546228 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 04:11:57.546382 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260926 04:11:57.547129 22335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4: No bootstrap required, opened a new log
I20260926 04:11:57.547621 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:57.549868 22335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b47fa9f00943be916f05345364b4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 04:11:57.550069 22335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:57.550117 22335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b47fa9f00943be916f05345364b4c4, State: Initialized, Role: FOLLOWER
I20260926 04:11:57.550247 22335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b47fa9f00943be916f05345364b4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 04:11:57.550362 22335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:57.550423 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:57.549241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.229:44893 (local address 127.0.0.1:35665)
0926 04:11:57.549301 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:57.549303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:57.549467 (+ 164us) server_negotiation.cc:407] Connection header received
0926 04:11:57.549518 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:57.549519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:57.549523 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:57.549574 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:57.550134 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:57.550135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:57.550135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:11:57.550139 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:57.550153 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:11:57.550157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:57.550212 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:57.550307 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 04:11:57.550320 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 04:11:57.550446 22335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:57.550601 22335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:57.551321 22335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b47fa9f00943be916f05345364b4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 04:11:57.551445 22335 leader_election.cc:302] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88b47fa9f00943be916f05345364b4c4; no voters:
I20260926 04:11:57.551616 22335 leader_election.cc:288] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:57.551653 22345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:57.551879 22335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:57.551904 22345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [term 1 LEADER]: Becoming Leader. State: Replica: 88b47fa9f00943be916f05345364b4c4, State: Running, Role: LEADER
I20260926 04:11:57.551987 22345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b47fa9f00943be916f05345364b4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 04:11:57.552388 22346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88b47fa9f00943be916f05345364b4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b47fa9f00943be916f05345364b4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260926 04:11:57.552482 22346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:57.552708 22353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:57.552687 22347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b47fa9f00943be916f05345364b4c4. Latest consensus state: current_term: 1 leader_uuid: "88b47fa9f00943be916f05345364b4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88b47fa9f00943be916f05345364b4c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260926 04:11:57.553128 22347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4 [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:57.553406 22353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:57.555177 22353 catalog_manager.cc:1347] Generated new cluster ID: 5420b13b5a064fc3ac6da8f6cad168a5
I20260926 04:11:57.555280 22353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:11:57.573267 22353 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:57.573465 22353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:57.587474 22353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88b47fa9f00943be916f05345364b4c4: Generated new TSK 0
I20260926 04:11:57.587702 22353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:57.598912 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22267
2026-09-26T04:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:57 dist-test-slave-xf24 krb5kdc[22366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:57 dist-test-slave-xf24 krb5kdc[22366](info): set up 2 sockets
Sep 26 04:11:57 dist-test-slave-xf24 krb5kdc[22366](info): commencing operation
krb5kdc: starting...
W20260926 04:11:58.700243 855 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:11:58 dist-test-slave-xf24 krb5kdc[22366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:11:58.819489 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45049
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:45049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:11:58.903249 22379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:11:58.903512 22379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:11:58.903553 22379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:11:58.904841 22379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:11:58.904944 22379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:11:58.904970 22379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:11:58.905023 22379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:11:58.905054 22379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:11:58.906606 22379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:45049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:11:58.906909 22379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:11:58.907152 22379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:11:58.912221 22387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:58.912225 22384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:11:58.912221 22385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:11:58.912252 22379 server_base.cc:1034] running on GCE node
I20260926 04:11:58.912750 22379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:11:58.913059 22379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:11:58.914227 22379 hybrid_clock.cc:648] HybridClock initialized: now 1790395918914208 us; error 40 us; skew 500 ppm
Sep 26 04:11:58 dist-test-slave-xf24 krb5kdc[22366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:11:58.921175 22379 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:11:58.921778 22379 webserver.cc:460] Webserver started at http://127.0.213.254:33021/ using document root <none> and password file <none>
I20260926 04:11:58.922037 22379 fs_manager.cc:359] Metadata directory not provided
I20260926 04:11:58.922111 22379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:11:58.922266 22379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:11:58.923380 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "99104f8b525b493592602d05e7db453d"
format_stamp: "Formatted at 2026-09-26 04:11:58 on dist-test-slave-xf24"
server_key: "98ab301c8cec74a262e7b6695de6db60"
server_key_iv: "583f421a4686dfc66f8bf88aaac5e1b9"
server_key_version: "encryptionkey@0"
I20260926 04:11:58.923763 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "99104f8b525b493592602d05e7db453d"
format_stamp: "Formatted at 2026-09-26 04:11:58 on dist-test-slave-xf24"
server_key: "98ab301c8cec74a262e7b6695de6db60"
server_key_iv: "583f421a4686dfc66f8bf88aaac5e1b9"
server_key_version: "encryptionkey@0"
I20260926 04:11:58.925992 22379 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 04:11:58.927088 22394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:11:58.927284 22379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:11:58.927413 22379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "99104f8b525b493592602d05e7db453d"
format_stamp: "Formatted at 2026-09-26 04:11:58 on dist-test-slave-xf24"
server_key: "98ab301c8cec74a262e7b6695de6db60"
server_key_iv: "583f421a4686dfc66f8bf88aaac5e1b9"
server_key_version: "encryptionkey@0"
I20260926 04:11:58.927518 22379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:11:58.958832 22379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:11:58.961086 22379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:11:58.961201 22379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:11:58.965288 22379 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:45049
I20260926 04:11:58.965361 22446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:45049 every 8 connection(s)
I20260926 04:11:58.965675 22379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:11:58.966572 22447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:11:58.968961 22447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d: Bootstrap starting.
I20260926 04:11:58.969548 22447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d: Neither blocks nor log segments found. Creating new log.
I20260926 04:11:58.969834 22447 log.cc:824] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:11:58.970558 22447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d: No bootstrap required, opened a new log
I20260926 04:11:58.972154 22447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99104f8b525b493592602d05e7db453d" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45049 } }
I20260926 04:11:58.972291 22447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:11:58.972314 22447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99104f8b525b493592602d05e7db453d, State: Initialized, Role: FOLLOWER
I20260926 04:11:58.972499 22447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99104f8b525b493592602d05e7db453d" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45049 } }
I20260926 04:11:58.972582 22447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:11:58.972625 22447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:11:58.972674 22447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:11:58.973223 22447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99104f8b525b493592602d05e7db453d" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45049 } }
I20260926 04:11:58.973311 22447 leader_election.cc:302] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99104f8b525b493592602d05e7db453d; no voters:
I20260926 04:11:58.973513 22447 leader_election.cc:288] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:11:58.973558 22450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:11:58.973773 22447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:11:58.973802 22450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [term 1 LEADER]: Becoming Leader. State: Replica: 99104f8b525b493592602d05e7db453d, State: Running, Role: LEADER
I20260926 04:11:58.973887 22450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99104f8b525b493592602d05e7db453d" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45049 } }
I20260926 04:11:58.974357 22452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99104f8b525b493592602d05e7db453d. Latest consensus state: current_term: 1 leader_uuid: "99104f8b525b493592602d05e7db453d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99104f8b525b493592602d05e7db453d" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45049 } } }
I20260926 04:11:58.974468 22452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:58.974572 22451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99104f8b525b493592602d05e7db453d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99104f8b525b493592602d05e7db453d" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 45049 } } }
I20260926 04:11:58.974881 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22379
I20260926 04:11:58.975003 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:11:58.975204 855 external_mini_cluster.cc:1428] Setting key b2811a36a6c65e8848cd9c4377ccf14a
I20260926 04:11:58.974714 22451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d [sys.catalog]: This master's current role is: LEADER
I20260926 04:11:58.974675 22459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:11:58.976363 22459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:11:58.977864 22459 catalog_manager.cc:1347] Generated new cluster ID: 6addd6375fdf419e942bf0e9df902577
I20260926 04:11:58.977916 22459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:11:58 dist-test-slave-xf24 krb5kdc[22366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:11:58.985348 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:58.975693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33071 (local address 127.0.213.254:45049)
0926 04:11:58.975850 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:58.975860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:58.975901 (+ 41us) server_negotiation.cc:407] Connection header received
0926 04:11:58.976032 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:58.976037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:58.976150 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:58.976312 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:58.976993 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:58.978137 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:58.978923 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:58.979349 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:58.979468 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:58.981943 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:58.981947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:58.981948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:58.981960 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:58.983663 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:58.984098 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:58.984101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:58.984102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:58.984207 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:58.984448 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:58.984449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:58.984449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:58.984689 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:58.984971 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:58.985062 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 04:11:58.985166 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260926 04:11:58.986316 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:11:58.987846 22459 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:11:58.988605 22459 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:11:58.995551 22459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99104f8b525b493592602d05e7db453d: Generated new TSK 0
I20260926 04:11:58.995783 22459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:11:58.999431 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:11:58.987684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56749 (local address 127.0.213.254:45049)
0926 04:11:58.987867 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 04:11:58.987869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:11:58.987879 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:11:58.992308 (+ 4429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:11:58.992310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:11:58.992326 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:11:58.992392 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:11:58.993076 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:58.993675 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:58.994519 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:11:58.994768 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:11:58.994867 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:11:58.996546 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:11:58.996547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:11:58.996548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:11:58.996556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:11:58.998078 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:58.998491 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:58.998493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:58.998493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:58.998566 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:11:58.998835 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:11:58.998836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:11:58.998837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:11:58.998963 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:11:58.999105 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 04:11:58.999241 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 04:11:58.999321 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260926 04:11:59.001582 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22379
W20260926 04:11:59.006122 22364 connection.cc:535] client connection to 127.0.213.254:45049 recv error: Network error: failed to read from TLS socket (remote: 127.0.213.254:45049): Connection reset by peer (error 104)
2026-09-26T04:11:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:11:59 dist-test-slave-xf24 krb5kdc[22478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:11:59 dist-test-slave-xf24 krb5kdc[22478](info): set up 2 sockets
Sep 26 04:11:59 dist-test-slave-xf24 krb5kdc[22478](info): commencing operation
krb5kdc: starting...
W20260926 04:12:00.032560 855 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:00 dist-test-slave-xf24 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:00.155412 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35849
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:35849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:00.239862 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:00.240120 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:00.240145 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:00.241396 22491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:00.241448 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:00.241463 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:00.241475 22491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:00.241487 22491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:00.243171 22491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:35849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:35849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:00.243446 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:00.243691 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:00.248963 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:00.249018 22491 server_base.cc:1034] running on GCE node
W20260926 04:12:00.248963 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:00.248963 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:00.249539 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:00.249912 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:00.251066 22491 hybrid_clock.cc:648] HybridClock initialized: now 1790395920251055 us; error 32 us; skew 500 ppm
Sep 26 04:12:00 dist-test-slave-xf24 krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:00.253681 22491 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:00.254153 22491 webserver.cc:460] Webserver started at http://127.0.213.254:39015/ using document root <none> and password file <none>
I20260926 04:12:00.254387 22491 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:00.254456 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:00.254607 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:00.255681 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "d125605de7564d799fbd21a9718239e2"
format_stamp: "Formatted at 2026-09-26 04:12:00 on dist-test-slave-xf24"
server_key: "829725dee642941758ed7745cf034d8c"
server_key_iv: "6b35fb0637841eba3267826feccc6518"
server_key_version: "encryptionkey@0"
I20260926 04:12:00.256038 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "d125605de7564d799fbd21a9718239e2"
format_stamp: "Formatted at 2026-09-26 04:12:00 on dist-test-slave-xf24"
server_key: "829725dee642941758ed7745cf034d8c"
server_key_iv: "6b35fb0637841eba3267826feccc6518"
server_key_version: "encryptionkey@0"
I20260926 04:12:00.257403 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:00.258138 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:00.258354 22491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:00.258464 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "d125605de7564d799fbd21a9718239e2"
format_stamp: "Formatted at 2026-09-26 04:12:00 on dist-test-slave-xf24"
server_key: "829725dee642941758ed7745cf034d8c"
server_key_iv: "6b35fb0637841eba3267826feccc6518"
server_key_version: "encryptionkey@0"
I20260926 04:12:00.258551 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:00.277256 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:00.279667 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:00.279835 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:00.283937 22491 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:35849
I20260926 04:12:00.283978 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:35849 every 8 connection(s)
I20260926 04:12:00.284368 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:00.285249 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:00.287525 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2: Bootstrap starting.
I20260926 04:12:00.288190 22559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:00.288480 22559 log.cc:824] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:00.289263 22559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2: No bootstrap required, opened a new log
I20260926 04:12:00.290553 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22491
I20260926 04:12:00.290831 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:00.291021 855 external_mini_cluster.cc:1428] Setting key a8bd0ff4cc68be3d72c75d6fe52967a6
I20260926 04:12:00.291100 22559 raft_consensus.cc:348] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d125605de7564d799fbd21a9718239e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35849 } }
I20260926 04:12:00.291270 22559 raft_consensus.cc:374] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:12:00.291318 22559 raft_consensus.cc:729] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d125605de7564d799fbd21a9718239e2, State: Initialized, Role: FOLLOWER
I20260926 04:12:00.291469 22559 consensus_queue.cc:260] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d125605de7564d799fbd21a9718239e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35849 } }
I20260926 04:12:00.291565 22559 raft_consensus.cc:388] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:12:00.291604 22559 raft_consensus.cc:482] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:12:00.291651 22559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:00.292829 22559 raft_consensus.cc:504] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d125605de7564d799fbd21a9718239e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35849 } }
I20260926 04:12:00.292946 22559 leader_election.cc:302] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d125605de7564d799fbd21a9718239e2; no voters:
I20260926 04:12:00.293097 22559 leader_election.cc:288] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:12:00.293133 22564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:00.293238 22564 raft_consensus.cc:686] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [term 1 LEADER]: Becoming Leader. State: Replica: d125605de7564d799fbd21a9718239e2, State: Running, Role: LEADER
I20260926 04:12:00.293329 22564 consensus_queue.cc:237] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d125605de7564d799fbd21a9718239e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35849 } }
I20260926 04:12:00.293370 22559 sys_catalog.cc:564] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:00.293645 22564 sys_catalog.cc:455] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d125605de7564d799fbd21a9718239e2. Latest consensus state: current_term: 1 leader_uuid: "d125605de7564d799fbd21a9718239e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d125605de7564d799fbd21a9718239e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35849 } } }
I20260926 04:12:00.293720 22564 sys_catalog.cc:458] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:00.293658 22565 sys_catalog.cc:455] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d125605de7564d799fbd21a9718239e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d125605de7564d799fbd21a9718239e2" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 35849 } } }
I20260926 04:12:00.293774 22565 sys_catalog.cc:458] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:00.294600 22568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:00.295096 22568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:00.296877 22568 catalog_manager.cc:1347] Generated new cluster ID: 879b3465b0d0428aabae06988a8c31c3
I20260926 04:12:00.296943 22568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:12:00.297240 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:00.291564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51235 (local address 127.0.213.254:35849)
0926 04:12:00.291742 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:00.291752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:00.291852 (+ 100us) server_negotiation.cc:407] Connection header received
0926 04:12:00.292002 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:00.292008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:00.292101 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:00.292222 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:00.292469 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:00.293608 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:00.294807 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:00.295487 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:00.295608 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:00.296470 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:00.296474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:00.296474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:00.296676 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:00.296715 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:00.296731 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:00.296832 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:00.297002 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 04:12:00.297033 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 04:12:00.298183 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:00.301280 22568 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:00.301980 22568 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:00.303350 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:00.299659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43931 (local address 127.0.213.254:35849)
0926 04:12:00.299693 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:00.299695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:00.299833 (+ 138us) server_negotiation.cc:407] Connection header received
0926 04:12:00.299862 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:00.299863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:00.299874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:00.299944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:00.300109 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:00.300862 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:00.301907 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:00.302286 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:00.302371 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:00.303089 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:00.303091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:00.303091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:00.303096 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:00.303114 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:00.303119 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:00.303187 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:00.303252 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 04:12:00.303266 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 04:12:00.306565 22568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d125605de7564d799fbd21a9718239e2: Generated new TSK 0
I20260926 04:12:00.306747 22568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:12:00.310184 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22491
W20260926 04:12:00.314910 22476 connection.cc:535] client connection to 127.0.213.254:35849 recv error: Network error: failed to read from TLS socket (remote: 127.0.213.254:35849): Connection reset by peer (error 104)
2026-09-26T04:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:12:00 dist-test-slave-xf24 krb5kdc[22590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:12:00 dist-test-slave-xf24 krb5kdc[22590](info): set up 2 sockets
Sep 26 04:12:00 dist-test-slave-xf24 krb5kdc[22590](info): commencing operation
krb5kdc: starting...
W20260926 04:12:01.364317 855 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:01 dist-test-slave-xf24 krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:01.491034 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34239
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:34239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:01.576117 22603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:01.576340 22603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:01.576364 22603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:01.577491 22603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:01.577528 22603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:01.577543 22603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:01.577556 22603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:01.577569 22603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:01.579003 22603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:34239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:01.579200 22603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:01.579392 22603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:01.584460 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:01.584453 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:01.584460 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:01.584498 22603 server_base.cc:1034] running on GCE node
I20260926 04:12:01.584971 22603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:01.585285 22603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:01.586454 22603 hybrid_clock.cc:648] HybridClock initialized: now 1790395921586429 us; error 51 us; skew 500 ppm
Sep 26 04:12:01 dist-test-slave-xf24 krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:01.589223 22603 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:01.589681 22603 webserver.cc:460] Webserver started at http://127.0.213.254:39161/ using document root <none> and password file <none>
I20260926 04:12:01.589937 22603 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:01.590006 22603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:01.590160 22603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:01.591244 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0"
format_stamp: "Formatted at 2026-09-26 04:12:01 on dist-test-slave-xf24"
server_key: "6506a1a15e24aacd95b498b0bbb65673"
server_key_iv: "3060f57f922a4cef34d1d8420ba230a1"
server_key_version: "encryptionkey@0"
I20260926 04:12:01.591624 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0"
format_stamp: "Formatted at 2026-09-26 04:12:01 on dist-test-slave-xf24"
server_key: "6506a1a15e24aacd95b498b0bbb65673"
server_key_iv: "3060f57f922a4cef34d1d8420ba230a1"
server_key_version: "encryptionkey@0"
I20260926 04:12:01.593186 22603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:12:01.593966 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:01.594211 22603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:12:01.594322 22603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0"
format_stamp: "Formatted at 2026-09-26 04:12:01 on dist-test-slave-xf24"
server_key: "6506a1a15e24aacd95b498b0bbb65673"
server_key_iv: "3060f57f922a4cef34d1d8420ba230a1"
server_key_version: "encryptionkey@0"
I20260926 04:12:01.594403 22603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:01.598038 22603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:01.600508 22603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:01.600680 22603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:01.604743 22603 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:34239
I20260926 04:12:01.604777 22670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:34239 every 8 connection(s)
I20260926 04:12:01.605122 22603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:01.606166 22671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:01.606298 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22603
I20260926 04:12:01.606442 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:01.606639 855 external_mini_cluster.cc:1428] Setting key 4f2c8b8b740e80e7bf9eb29a919c7c59
I20260926 04:12:01.609445 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:01.607207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41997 (local address 127.0.213.254:34239)
0926 04:12:01.607405 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:01.607415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:01.607457 (+ 42us) server_negotiation.cc:407] Connection header received
0926 04:12:01.607942 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:01.607952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:01.607992 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:01.608189 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:01.608899 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:01.608903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:01.608904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:01.608909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:01.608955 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:01.608970 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:01.609046 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:01.609195 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 04:12:01.609246 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260926 04:12:01.609972 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0: Bootstrap starting.
I20260926 04:12:01.610368 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:01.610749 22671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:01.611047 22671 log.cc:824] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:01.611874 22671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0: No bootstrap required, opened a new log
I20260926 04:12:01.613279 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:01.611885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36823 (local address 127.0.213.254:34239)
0926 04:12:01.611924 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:01.611928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:01.612161 (+ 233us) server_negotiation.cc:407] Connection header received
0926 04:12:01.612240 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:01.612241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:01.612246 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:01.612301 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:01.613038 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:01.613039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:01.613040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:01.613044 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:01.613060 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:01.613064 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:01.613138 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:01.613198 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 04:12:01.613213 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 04:12:01.614238 22671 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34239 } }
I20260926 04:12:01.614390 22671 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:12:01.614414 22671 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d9ef336e7b4645a83c7550b1b0d9d0, State: Initialized, Role: FOLLOWER
I20260926 04:12:01.614504 22671 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34239 } }
I20260926 04:12:01.614568 22671 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:12:01.614589 22671 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:12:01.614617 22671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:01.615154 22671 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34239 } }
I20260926 04:12:01.615245 22671 leader_election.cc:302] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8d9ef336e7b4645a83c7550b1b0d9d0; no voters:
I20260926 04:12:01.615388 22671 leader_election.cc:288] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:12:01.615447 22681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:01.615622 22681 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [term 1 LEADER]: Becoming Leader. State: Replica: a8d9ef336e7b4645a83c7550b1b0d9d0, State: Running, Role: LEADER
I20260926 04:12:01.615675 22671 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:01.615778 22681 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34239 } }
I20260926 04:12:01.616190 22683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8d9ef336e7b4645a83c7550b1b0d9d0. Latest consensus state: current_term: 1 leader_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34239 } } }
I20260926 04:12:01.616189 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8d9ef336e7b4645a83c7550b1b0d9d0" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34239 } } }
I20260926 04:12:01.616312 22683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:01.616317 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:01.616771 22689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:01.617319 22689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:01.618876 22689 catalog_manager.cc:1347] Generated new cluster ID: f3bd656977034596bb7eb665e4a8bac2
I20260926 04:12:01.618988 22689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:12:01.635531 22689 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:01.635752 22689 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:01.651849 22689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8d9ef336e7b4645a83c7550b1b0d9d0: Generated new TSK 0
I20260926 04:12:01.652150 22689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:12:01.663429 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22603
2026-09-26T04:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:12:01 dist-test-slave-xf24 krb5kdc[22702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:12:01 dist-test-slave-xf24 krb5kdc[22702](info): set up 2 sockets
Sep 26 04:12:01 dist-test-slave-xf24 krb5kdc[22702](info): commencing operation
krb5kdc: starting...
W20260926 04:12:02.704901 855 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:02 dist-test-slave-xf24 krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:02.833583 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34421
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:34421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:02.917210 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:02.917425 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:02.917449 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:02.918735 22715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:02.918795 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:02.918812 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:02.918824 22715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:02.918836 22715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:02.920415 22715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:34421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:34421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:02.920691 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:02.920938 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:02.926577 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:02.926585 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:02.926743 22715 server_base.cc:1034] running on GCE node
W20260926 04:12:02.926750 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:02.927145 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:02.927517 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:02.928671 22715 hybrid_clock.cc:648] HybridClock initialized: now 1790395922928648 us; error 42 us; skew 500 ppm
Sep 26 04:12:02 dist-test-slave-xf24 krb5kdc[22702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:02.932191 22715 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:02.932744 22715 webserver.cc:460] Webserver started at http://127.0.213.254:38763/ using document root <none> and password file <none>
I20260926 04:12:02.933017 22715 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:02.933089 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:02.933235 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:02.934324 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "2816f14aa5b04ad1b3486f31c8042d2f"
format_stamp: "Formatted at 2026-09-26 04:12:02 on dist-test-slave-xf24"
server_key: "e5eba1f42d078a89830fdb5623f03a13"
server_key_iv: "9233372a756c31744467773c161f7994"
server_key_version: "encryptionkey@0"
I20260926 04:12:02.934690 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "2816f14aa5b04ad1b3486f31c8042d2f"
format_stamp: "Formatted at 2026-09-26 04:12:02 on dist-test-slave-xf24"
server_key: "e5eba1f42d078a89830fdb5623f03a13"
server_key_iv: "9233372a756c31744467773c161f7994"
server_key_version: "encryptionkey@0"
I20260926 04:12:02.936210 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:12:02.937069 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:02.937285 22715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:02.937394 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "2816f14aa5b04ad1b3486f31c8042d2f"
format_stamp: "Formatted at 2026-09-26 04:12:02 on dist-test-slave-xf24"
server_key: "e5eba1f42d078a89830fdb5623f03a13"
server_key_iv: "9233372a756c31744467773c161f7994"
server_key_version: "encryptionkey@0"
I20260926 04:12:02.937496 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:02.955592 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:02.958092 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:02.958268 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:02.962541 22715 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:34421
I20260926 04:12:02.962582 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:34421 every 8 connection(s)
I20260926 04:12:02.962965 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:02.964133 22783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:02.966530 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f: Bootstrap starting.
I20260926 04:12:02.967119 22783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:02.967417 22783 log.cc:824] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:02.968209 22783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f: No bootstrap required, opened a new log
I20260926 04:12:02.969129 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22715
I20260926 04:12:02.969215 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:02.969429 855 external_mini_cluster.cc:1428] Setting key cfc18bde072da0a3a925f17c09da1039
I20260926 04:12:02.970207 22783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34421 } }
I20260926 04:12:02.970407 22783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:12:02.970466 22783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2816f14aa5b04ad1b3486f31c8042d2f, State: Initialized, Role: FOLLOWER
I20260926 04:12:02.970600 22783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34421 } }
I20260926 04:12:02.970682 22783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:12:02.970726 22783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:12:02.970777 22783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:02.971635 22783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34421 } }
I20260926 04:12:02.971760 22783 leader_election.cc:302] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2816f14aa5b04ad1b3486f31c8042d2f; no voters:
I20260926 04:12:02.971922 22783 leader_election.cc:288] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:12:02.971974 22788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:02.972154 22788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [term 1 LEADER]: Becoming Leader. State: Replica: 2816f14aa5b04ad1b3486f31c8042d2f, State: Running, Role: LEADER
I20260926 04:12:02.972229 22783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:02.972241 22788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34421 } }
I20260926 04:12:02.972731 22790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2816f14aa5b04ad1b3486f31c8042d2f. Latest consensus state: current_term: 1 leader_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34421 } } }
I20260926 04:12:02.972715 22789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2816f14aa5b04ad1b3486f31c8042d2f" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 34421 } } }
I20260926 04:12:02.972821 22790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:02.972828 22789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:02.973049 22797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:02.973538 22797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:02.975378 22797 catalog_manager.cc:1347] Generated new cluster ID: a4d388de5a164b469a22c2843a6934a6
I20260926 04:12:02.975447 22797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:12:02 dist-test-slave-xf24 krb5kdc[22702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:12:02.983819 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:02.970077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41799 (local address 127.0.213.254:34421)
0926 04:12:02.970264 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:02.970274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:02.973971 (+ 3697us) server_negotiation.cc:407] Connection header received
0926 04:12:02.974335 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:02.974344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:02.974423 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:02.974586 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:02.975352 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:02.976463 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:02.977278 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:02.977741 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:02.977870 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:02.980425 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:02.980430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:02.980431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:02.980447 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:02.982208 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:02.982610 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:02.982613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:02.982613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:02.982722 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:02.982944 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:02.982945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:02.982945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:02.983204 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:02.983396 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:02.983474 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 04:12:02.983624 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260926 04:12:02.984825 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:02.989209 22797 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:02.989912 22797 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:02.993150 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:02.986285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33683 (local address 127.0.213.254:34421)
0926 04:12:02.986325 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:02.986327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:02.986463 (+ 136us) server_negotiation.cc:407] Connection header received
0926 04:12:02.986535 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:02.986536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:02.986547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:02.986601 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:02.987161 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:02.987674 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:02.988621 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:02.988967 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:02.989045 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:02.990397 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:02.990399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:02.990399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:02.990407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:02.991867 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:02.992299 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:02.992300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:02.992300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:02.992362 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:02.992592 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:02.992593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:02.992593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:02.992707 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:02.992827 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:02.992950 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 04:12:02.993042 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:12:02.994587 22797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2816f14aa5b04ad1b3486f31c8042d2f: Generated new TSK 0
I20260926 04:12:02.994786 22797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:12:02.997766 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22715
2026-09-26T04:12:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:12:03 dist-test-slave-xf24 krb5kdc[22814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:12:03 dist-test-slave-xf24 krb5kdc[22814](info): set up 2 sockets
Sep 26 04:12:03 dist-test-slave-xf24 krb5kdc[22814](info): commencing operation
krb5kdc: starting...
W20260926 04:12:04.040520 855 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:04 dist-test-slave-xf24 krb5kdc[22814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:04.176481 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36361
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:36361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:04.262127 22827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:04.262400 22827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:04.262445 22827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:04.263798 22827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:04.263927 22827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:04.263958 22827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:04.263991 22827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:04.264021 22827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:04.265745 22827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:36361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:04.266058 22827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:04.266299 22827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:04.271628 22832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:04.271628 22835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:04.271621 22833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:04.272177 22827 server_base.cc:1034] running on GCE node
I20260926 04:12:04.272480 22827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:04.272809 22827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:04.274001 22827 hybrid_clock.cc:648] HybridClock initialized: now 1790395924273989 us; error 42 us; skew 500 ppm
Sep 26 04:12:04 dist-test-slave-xf24 krb5kdc[22814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:04.280725 22827 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:04.281260 22827 webserver.cc:460] Webserver started at http://127.0.213.254:35237/ using document root <none> and password file <none>
I20260926 04:12:04.281555 22827 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:04.281639 22827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:04.281796 22827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:04.282939 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "4e0371fe69ce42f2a3c2470ab6ee472e"
format_stamp: "Formatted at 2026-09-26 04:12:04 on dist-test-slave-xf24"
server_key: "1ccf6899c2aca6a1d8a92a3314e978f3"
server_key_iv: "2ea62b17884753714a02ecc4dfed2aa3"
server_key_version: "encryptionkey@0"
I20260926 04:12:04.283366 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "4e0371fe69ce42f2a3c2470ab6ee472e"
format_stamp: "Formatted at 2026-09-26 04:12:04 on dist-test-slave-xf24"
server_key: "1ccf6899c2aca6a1d8a92a3314e978f3"
server_key_iv: "2ea62b17884753714a02ecc4dfed2aa3"
server_key_version: "encryptionkey@0"
I20260926 04:12:04.284916 22827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:12:04.285728 22842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:04.285913 22827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:12:04.286007 22827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "4e0371fe69ce42f2a3c2470ab6ee472e"
format_stamp: "Formatted at 2026-09-26 04:12:04 on dist-test-slave-xf24"
server_key: "1ccf6899c2aca6a1d8a92a3314e978f3"
server_key_iv: "2ea62b17884753714a02ecc4dfed2aa3"
server_key_version: "encryptionkey@0"
I20260926 04:12:04.286094 22827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:04.296269 22827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:04.298867 22827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:04.299073 22827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:04.303251 22827 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:36361
I20260926 04:12:04.303289 22894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:36361 every 8 connection(s)
I20260926 04:12:04.303661 22827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:04.304661 22895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:04.306937 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e: Bootstrap starting.
I20260926 04:12:04.307556 22895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:04.307855 22895 log.cc:824] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:04.308696 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e: No bootstrap required, opened a new log
I20260926 04:12:04.310380 22895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36361 } }
I20260926 04:12:04.310534 22895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:12:04.310556 22895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0371fe69ce42f2a3c2470ab6ee472e, State: Initialized, Role: FOLLOWER
I20260926 04:12:04.310760 22895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36361 } }
I20260926 04:12:04.310845 22895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:12:04.310870 22895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:12:04.310900 22895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:04.311439 22895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36361 } }
I20260926 04:12:04.311530 22895 leader_election.cc:302] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e0371fe69ce42f2a3c2470ab6ee472e; no voters:
I20260926 04:12:04.311697 22895 leader_election.cc:288] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:12:04.311726 22898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:04.311846 22898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [term 1 LEADER]: Becoming Leader. State: Replica: 4e0371fe69ce42f2a3c2470ab6ee472e, State: Running, Role: LEADER
I20260926 04:12:04.312004 22895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:04.312114 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22827
I20260926 04:12:04.312224 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:04.312417 855 external_mini_cluster.cc:1428] Setting key 36e542b3e8868c8bf28300193ec352d9
I20260926 04:12:04.312598 22898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36361 } }
I20260926 04:12:04.313318 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36361 } } }
I20260926 04:12:04.313416 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:04.313755 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e0371fe69ce42f2a3c2470ab6ee472e. Latest consensus state: current_term: 1 leader_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e0371fe69ce42f2a3c2470ab6ee472e" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 36361 } } }
W20260926 04:12:04.314108 22915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 04:12:04.314149 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e [sys.catalog]: This master's current role is: LEADER
W20260926 04:12:04.314216 22915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:04.315845 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:04.316249 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:04.318337 22914 catalog_manager.cc:1347] Generated new cluster ID: 9959a6f0e76744fe80247a765585685f
I20260926 04:12:04.318403 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:12:04.319231 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:04.312954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34831 (local address 127.0.213.254:36361)
0926 04:12:04.313342 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:04.313354 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:04.313399 (+ 45us) server_negotiation.cc:407] Connection header received
0926 04:12:04.313555 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:04.313563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:04.313630 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:04.314080 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:04.314178 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:04.315655 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:04.316824 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:04.317487 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:04.317624 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:04.318489 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:04.318493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:04.318493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:04.318705 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:04.318745 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:04.318760 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:04.318839 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:04.318946 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 04:12:04.319012 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":61,"threads_started":1}
I20260926 04:12:04.320188 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:04.325297 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:04.321661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49551 (local address 127.0.213.254:36361)
0926 04:12:04.321695 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:04.321697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:04.321825 (+ 128us) server_negotiation.cc:407] Connection header received
0926 04:12:04.321889 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:04.321890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:04.321901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:04.321955 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:04.322115 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:04.322848 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:04.323902 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:04.324227 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:04.324293 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:04.325037 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:04.325038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:04.325039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:04.325044 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:04.325059 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:04.325063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:04.325115 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:04.325181 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 04:12:04.325210 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 04:12:04.327510 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:04.328137 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:04.331305 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e0371fe69ce42f2a3c2470ab6ee472e: Generated new TSK 0
I20260926 04:12:04.331537 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:12:04.336164 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22827
2026-09-26T04:12:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:12:04 dist-test-slave-xf24 krb5kdc[22926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:12:04 dist-test-slave-xf24 krb5kdc[22926](info): set up 2 sockets
Sep 26 04:12:04 dist-test-slave-xf24 krb5kdc[22926](info): commencing operation
krb5kdc: starting...
W20260926 04:12:05.412384 855 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:05 dist-test-slave-xf24 krb5kdc[22926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:05.542867 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:38947
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:45853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:38947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:05.625912 22939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:05.626176 22939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:05.626220 22939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:05.627446 22939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:05.627522 22939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:05.627547 22939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:05.627569 22939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:05.627597 22939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:05.629159 22939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:45853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:38947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:05.629463 22939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:05.629701 22939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:05.635046 22945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:05.635036 22947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:05.635033 22944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:05.635358 22939 server_base.cc:1034] running on GCE node
I20260926 04:12:05.635619 22939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:05.635933 22939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:05.637176 22939 hybrid_clock.cc:648] HybridClock initialized: now 1790395925637162 us; error 36 us; skew 500 ppm
Sep 26 04:12:05 dist-test-slave-xf24 krb5kdc[22926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:05.640132 22939 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:05.640627 22939 webserver.cc:460] Webserver started at http://127.0.213.254:33017/ using document root <none> and password file <none>
I20260926 04:12:05.640884 22939 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:05.640952 22939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:05.641106 22939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:05.642179 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "5742fc6523b04ccea0f7cc3ec29813f4"
format_stamp: "Formatted at 2026-09-26 04:12:05 on dist-test-slave-xf24"
server_key: "29d59a3ce532f01a6b520d3a9e5d30e9"
server_key_iv: "d3561e75d06b2e3270e4d030da4abcc5"
server_key_version: "encryptionkey@0"
I20260926 04:12:05.642556 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "5742fc6523b04ccea0f7cc3ec29813f4"
format_stamp: "Formatted at 2026-09-26 04:12:05 on dist-test-slave-xf24"
server_key: "29d59a3ce532f01a6b520d3a9e5d30e9"
server_key_iv: "d3561e75d06b2e3270e4d030da4abcc5"
server_key_version: "encryptionkey@0"
I20260926 04:12:05.643976 22939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:12:05.644886 22954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:05.645109 22939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:05.645228 22939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "5742fc6523b04ccea0f7cc3ec29813f4"
format_stamp: "Formatted at 2026-09-26 04:12:05 on dist-test-slave-xf24"
server_key: "29d59a3ce532f01a6b520d3a9e5d30e9"
server_key_iv: "d3561e75d06b2e3270e4d030da4abcc5"
server_key_version: "encryptionkey@0"
I20260926 04:12:05.645310 22939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:05.648929 22939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:05.651278 22939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:05.651438 22939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:05.655689 22939 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:38947
I20260926 04:12:05.655727 23006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:38947 every 8 connection(s)
I20260926 04:12:05.656141 22939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:05.657042 23007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:05.658010 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 22939
I20260926 04:12:05.658123 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:05.658299 855 external_mini_cluster.cc:1428] Setting key 03ffb016cf18da3041782710b4771ac3
I20260926 04:12:05.660712 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4: Bootstrap starting.
I20260926 04:12:05.661422 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:05.658983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49695 (local address 127.0.213.254:38947)
0926 04:12:05.659219 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:05.659229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:05.659262 (+ 33us) server_negotiation.cc:407] Connection header received
0926 04:12:05.659809 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:05.659819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:05.659859 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:05.660024 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:05.660926 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:05.660929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:05.660929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:05.660933 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:05.660973 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:05.660984 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:05.661053 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:05.661156 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 04:12:05.661189 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260926 04:12:05.661445 23007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:05.661818 23007 log.cc:824] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:05.662544 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:05.662672 23007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4: No bootstrap required, opened a new log
I20260926 04:12:05.665010 23007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38947 } }
I20260926 04:12:05.665220 23007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:12:05.665287 23007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5742fc6523b04ccea0f7cc3ec29813f4, State: Initialized, Role: FOLLOWER
I20260926 04:12:05.665494 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:05.664005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47567 (local address 127.0.213.254:38947)
0926 04:12:05.664049 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:05.664074 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:05.664227 (+ 153us) server_negotiation.cc:407] Connection header received
0926 04:12:05.664302 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:05.664303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:05.664307 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:05.664353 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:05.665157 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:05.665158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:05.665159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:12:05.665163 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:05.665177 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:12:05.665181 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:05.665296 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:05.665380 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 04:12:05.665393 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:12:05.665422 23007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38947 } }
I20260926 04:12:05.665593 23007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:12:05.665640 23007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:12:05.665715 23007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:05.666399 23007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38947 } }
I20260926 04:12:05.666495 23007 leader_election.cc:302] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5742fc6523b04ccea0f7cc3ec29813f4; no voters:
I20260926 04:12:05.666728 23007 leader_election.cc:288] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:12:05.666919 23017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:05.667001 23007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:05.667150 23017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [term 1 LEADER]: Becoming Leader. State: Replica: 5742fc6523b04ccea0f7cc3ec29813f4, State: Running, Role: LEADER
I20260926 04:12:05.667353 23017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38947 } }
I20260926 04:12:05.667825 23019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5742fc6523b04ccea0f7cc3ec29813f4. Latest consensus state: current_term: 1 leader_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38947 } } }
I20260926 04:12:05.667912 23019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:05.668082 23018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5742fc6523b04ccea0f7cc3ec29813f4" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 38947 } } }
I20260926 04:12:05.668146 23018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:05.668305 23027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:05.668860 23027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:05.670715 23027 catalog_manager.cc:1347] Generated new cluster ID: ca07abbbaef7489eab856e773306ec27
I20260926 04:12:05.670786 23027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:12:05.684978 23027 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:05.685214 23027 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:05.688864 23027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5742fc6523b04ccea0f7cc3ec29813f4: Generated new TSK 0
I20260926 04:12:05.689116 23027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:12:05.697835 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 22939
2026-09-26T04:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1360 ms)
[----------] 18 tests from AuthTokenIssuingTest (24331 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:12:05 dist-test-slave-xf24 krb5kdc[23038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:12:05 dist-test-slave-xf24 krb5kdc[23038](info): set up 2 sockets
Sep 26 04:12:05 dist-test-slave-xf24 krb5kdc[23038](info): commencing operation
krb5kdc: starting...
W20260926 04:12:06.772826 855 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:06 dist-test-slave-xf24 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:06.899909 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:42379
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:42379,127.0.213.253:33459,127.0.213.252:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:06.986950 23051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:12:06.987272 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:06.987311 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:06.987360 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:06.989526 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:06.989603 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:06.989624 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:06.989642 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:06.989660 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:06.992443 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:42379,127.0.213.253:33459,127.0.213.252:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:06.992786 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:06.993110 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:06.998335 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:06.998314 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:06.998562 23051 server_base.cc:1034] running on GCE node
W20260926 04:12:06.998363 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:06.998867 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:06.999176 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:07.000377 23051 hybrid_clock.cc:648] HybridClock initialized: now 1790395927000362 us; error 34 us; skew 500 ppm
Sep 26 04:12:06 dist-test-slave-xf24 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:07.003108 23051 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:07.003710 23051 webserver.cc:460] Webserver started at http://127.0.213.254:45505/ using document root <none> and password file <none>
I20260926 04:12:07.003907 23051 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:07.003950 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:07.004043 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:07.005148 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "d4717b254a5d412f9683bef7491a2466"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "c8b82dcf178201139806988fac480405"
server_key_iv: "ae92e0d2cdc80d746ea6afcf342d672e"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.005481 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "d4717b254a5d412f9683bef7491a2466"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "c8b82dcf178201139806988fac480405"
server_key_iv: "ae92e0d2cdc80d746ea6afcf342d672e"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.006819 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:12:07.007581 23066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:07.007783 23051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:07.007877 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "d4717b254a5d412f9683bef7491a2466"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "c8b82dcf178201139806988fac480405"
server_key_iv: "ae92e0d2cdc80d746ea6afcf342d672e"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.007936 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:07.021576 23051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:07.023640 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:07.023757 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:07.027690 23051 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:42379
I20260926 04:12:07.027750 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:42379 every 8 connection(s)
I20260926 04:12:07.028143 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:07.029220 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:07.030802 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.035281 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 23051
I20260926 04:12:07.035379 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:07.035578 855 external_mini_cluster.cc:1428] Setting key e29207e53da82b39b22cb2a586622e2f
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:12:07.041268 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.031584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.213.254:42379)
0926 04:12:07.031821 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.031831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.031962 (+ 131us) server_negotiation.cc:407] Connection header received
0926 04:12:07.032643 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.032645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.032663 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.032759 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.033326 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.034154 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.035051 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.035390 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.035490 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.037848 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.037855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.037855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.037862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.039547 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.039962 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.039966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.039966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.040106 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.040337 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.040338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.040339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.040570 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.040664 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.040863 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.041009 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260926 04:12:07.041265 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.031579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42379 (local address 127.0.0.1:35050)
0926 04:12:07.031809 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.031856 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.032462 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.032765 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.032768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.033063 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.033219 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.033223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.034254 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.034255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.034949 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.034951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.035504 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.035505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.035572 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.036239 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.036248 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.037760 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.039654 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.039661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.039663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.039865 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.040175 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.040176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.040176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.040270 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.040642 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.040648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.040711 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.040819 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.041073 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":104,"spinlock_wait_cycles":10624,"thread_start_us":102,"threads_started":1}
I20260926 04:12:07.042043 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.042484 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.042324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49616)
0926 04:12:07.042390 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.042448 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 04:12:07.042696 23070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.213.253@KRBTEST.COM; defaulting to no policy
I20260926 04:12:07.043303 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.043187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49618)
0926 04:12:07.043240 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.043272 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 04:12:07.043465 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:33459: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab.
Entry for principal kudu/127.0.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab.
I20260926 04:12:07.062651 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:33459
--webserver_interface=127.0.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab
--principal=kudu/127.0.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:42379,127.0.213.253:33459,127.0.213.252:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:12:07.092720 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } attempt: 1
I20260926 04:12:07.093350 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.093116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49630)
0926 04:12:07.093185 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.093271 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 04:12:07.093936 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.093780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49646)
0926 04:12:07.093844 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.093880 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 04:12:07.094130 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:33459: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
W20260926 04:12:07.148890 23126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:12:07.149119 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:07.149137 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:07.149158 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:07.150282 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:07.150318 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:07.150333 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:07.150345 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:07.150357 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:07.151839 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:42379,127.0.213.253:33459,127.0.213.252:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab
--principal=kudu/127.0.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:33459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:07.152051 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:07.152316 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:12:07.157800 23126 server_base.cc:1034] running on GCE node
W20260926 04:12:07.157763 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:07.157763 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:07.157758 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:07.158308 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:07.158643 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:07.159832 23126 hybrid_clock.cc:648] HybridClock initialized: now 1790395927159826 us; error 47 us; skew 500 ppm
I20260926 04:12:07.162333 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } attempt: 2
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:07.162760 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.162626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49652)
0926 04:12:07.162689 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.162722 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 04:12:07.163154 23126 init.cc:377] Logged in from keytab as kudu/127.0.213.253@KRBTEST.COM (short username kudu)
I20260926 04:12:07.163362 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.163180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49664)
0926 04:12:07.163288 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.163325 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 04:12:07.163470 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:33459: Network error: Client connection negotiation failed: client connection to 127.0.213.253:33459: connect: Connection refused (error 111)
I20260926 04:12:07.163753 23126 webserver.cc:460] Webserver started at http://127.0.213.253:40471/ using document root <none> and password file <none>
I20260926 04:12:07.164005 23126 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:07.164094 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:07.164238 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:07.165390 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data/instance:
uuid: "601f6f726c9a436a8032e1b316415a32"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "1a77cb168891c576f0dcfa1b78c46090"
server_key_iv: "ae2fbf468d8555a1b08b6cb71f9bb8a6"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.165741 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal/instance:
uuid: "601f6f726c9a436a8032e1b316415a32"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "1a77cb168891c576f0dcfa1b78c46090"
server_key_iv: "ae2fbf468d8555a1b08b6cb71f9bb8a6"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.167111 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:12:07.168006 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:07.168251 23126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:07.168352 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal
uuid: "601f6f726c9a436a8032e1b316415a32"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "1a77cb168891c576f0dcfa1b78c46090"
server_key_iv: "ae2fbf468d8555a1b08b6cb71f9bb8a6"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.168421 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:07.177721 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:07.179662 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:07.179773 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:07.183741 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.253:33459
I20260926 04:12:07.183815 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.253:33459 every 8 connection(s)
I20260926 04:12:07.184226 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
I20260926 04:12:07.185387 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:07.187049 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.187901 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 23126
I20260926 04:12:07.188036 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-1/wal/instance
I20260926 04:12:07.188261 855 external_mini_cluster.cc:1428] Setting key 305de13ca2bbef5cdaf6d03152ee4aba
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.252@KRBTEST.COM; defaulting to no policy
I20260926 04:12:07.198064 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.187813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.0.213.254:42379)
0926 04:12:07.187890 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.187893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.188172 (+ 279us) server_negotiation.cc:407] Connection header received
0926 04:12:07.189011 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.189013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.189026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.189082 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.190003 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.190814 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.192013 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.192383 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.192459 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.194872 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.194874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.194874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.194883 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.196615 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.197028 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.197029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.197029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.197086 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.197388 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.197388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.197388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.197529 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.197585 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.197913 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.197988 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 04:12:07.198344 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.187779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42379 (local address 127.0.0.1:35052)
0926 04:12:07.188005 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.188085 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.188866 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.189157 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.189162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.189569 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.189877 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.189884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.190995 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.190996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.191901 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.191904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.192499 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.192500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.192623 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.193223 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.193233 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.194762 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.196697 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.196705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.196707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.196951 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.197147 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.197148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.197148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.197322 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.197624 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.197631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.197775 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.197884 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.198063 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.252@KRBTEST.COM" created.
I20260926 04:12:07.199107 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:07.206920 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.199428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49676)
0926 04:12:07.199483 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.199495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.199541 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.199806 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.199806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.199945 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.199998 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.200000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.200896 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.200897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.201403 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.201405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.201916 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.201917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.201980 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.202478 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.202484 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.203985 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.205816 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.205817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.205818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.205979 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.206224 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.206225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.206225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.206278 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.206598 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.206598 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.206637 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.206699 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.206801 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 04:12:07.206954 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.199495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.213.253:33459)
0926 04:12:07.199663 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.199672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.199687 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:12:07.199703 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.199705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.199716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.199792 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.200109 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.200797 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.201493 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.201823 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.201916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.204100 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.204111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.204111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.204119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.205686 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.206095 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.206099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.206099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.206155 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.206347 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.206348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.206349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.206517 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.206609 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.206726 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.206853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260926 04:12:07.207743 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.208246 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.208137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33770)
0926 04:12:07.208172 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.208223 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 04:12:07.208392 23144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
I20260926 04:12:07.208879 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.208780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33772)
0926 04:12:07.208831 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.208855 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 04:12:07.209002 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:33387: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab.
Entry for principal kudu/127.0.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab.
I20260926 04:12:07.215418 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:33387
--webserver_interface=127.0.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab
--principal=kudu/127.0.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:42379,127.0.213.253:33459,127.0.213.252:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:12:07.254742 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } attempt: 3
I20260926 04:12:07.258229 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } attempt: 1
I20260926 04:12:07.258641 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.258516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33788)
0926 04:12:07.258569 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.258607 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 04:12:07.259006 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.258880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33804)
0926 04:12:07.258937 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.258974 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 04:12:07.259109 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:33387: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:07.262372 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.255183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49680)
0926 04:12:07.255249 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.255262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.255331 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.255484 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.255485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.255677 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.255747 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.255749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.256506 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.256507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.257046 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.257048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.257447 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.257448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.257535 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.258025 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.258031 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.259531 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.261256 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.261258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.261258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.261439 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.261671 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.261672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.261672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.261733 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.262005 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.262006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.262058 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.262124 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.262236 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 04:12:07.262403 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.255283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.213.253:33459)
0926 04:12:07.255359 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.255361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.255373 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:12:07.255406 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.255407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.255415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.255478 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.255839 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.256411 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.257137 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.257361 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.257449 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.259627 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.259629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.259629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.259637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.261125 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.261525 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.261526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.261527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.261594 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.261807 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.261807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.261808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.261931 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.262003 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.262212 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.262317 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:12:07.263240 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.263648 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.263533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33812)
0926 04:12:07.263584 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.263614 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 04:12:07.263964 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.263863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33828)
0926 04:12:07.263920 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.263944 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 04:12:07.264084 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:33387: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
I20260926 04:12:07.295274 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } attempt: 1
I20260926 04:12:07.295885 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.295739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33844)
0926 04:12:07.295813 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.295855 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 04:12:07.296249 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.296150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33848)
0926 04:12:07.296199 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.296231 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:12:07.296368 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:33387: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
W20260926 04:12:07.312152 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:12:07.312430 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:07.312462 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:07.312518 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:07.314018 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:07.314119 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:07.314147 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:07.314170 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:07.314200 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:07.315897 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:42379,127.0.213.253:33459,127.0.213.252:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab
--principal=kudu/127.0.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:07.316274 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:07.316537 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:07.321645 23206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:07.321681 23207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:07.321938 23209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:07.321956 23201 server_base.cc:1034] running on GCE node
I20260926 04:12:07.322324 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:07.322716 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:07.323886 23201 hybrid_clock.cc:648] HybridClock initialized: now 1790395927323859 us; error 78 us; skew 500 ppm
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:07.326886 23201 init.cc:377] Logged in from keytab as kudu/127.0.213.252@KRBTEST.COM (short username kudu)
I20260926 04:12:07.327267 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } attempt: 2
I20260926 04:12:07.327513 23201 webserver.cc:460] Webserver started at http://127.0.213.252:42031/ using document root <none> and password file <none>
I20260926 04:12:07.327689 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.327563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33858)
0926 04:12:07.327622 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.327662 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 04:12:07.327826 23201 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:07.327889 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:07.328037 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:07.328087 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.327968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33868)
0926 04:12:07.328000 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.328025 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 04:12:07.328198 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:33387: Network error: Client connection negotiation failed: client connection to 127.0.213.252:33387: connect: Connection refused (error 111)
I20260926 04:12:07.329209 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data/instance:
uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "795c9da6e889a61e350029795d838e65"
server_key_iv: "baa2c19efe33f54b33cb6258005dd43b"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.329591 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal/instance:
uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "795c9da6e889a61e350029795d838e65"
server_key_iv: "baa2c19efe33f54b33cb6258005dd43b"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.331218 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:12:07.332266 23216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:07.332489 23201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:07.332618 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal
uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc"
format_stamp: "Formatted at 2026-09-26 04:12:07 on dist-test-slave-xf24"
server_key: "795c9da6e889a61e350029795d838e65"
server_key_iv: "baa2c19efe33f54b33cb6258005dd43b"
server_key_version: "encryptionkey@0"
I20260926 04:12:07.332715 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:07.349625 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:07.352380 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:07.352574 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:07.356910 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.252:33387
I20260926 04:12:07.356953 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.252:33387 every 8 connection(s)
I20260926 04:12:07.357328 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
I20260926 04:12:07.358405 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:07.360172 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.361295 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 23201
I20260926 04:12:07.361407 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395654118158-855-0/minicluster-data/master-2/wal/instance
I20260926 04:12:07.361629 855 external_mini_cluster.cc:1428] Setting key 5376b78cc2a38c341f2a035377a9a44f
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
I20260926 04:12:07.371515 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } attempt: 2
I20260926 04:12:07.371941 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.360921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:42379 (local address 127.0.0.1:35068)
0926 04:12:07.361182 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.361222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.362016 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.362396 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.362401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.362829 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.363230 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.363236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.364238 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.364239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.365239 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.365242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.365692 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.365693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.365792 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.366393 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.366405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.368237 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.370131 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.370137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.370139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.370421 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.370637 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.370638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.370639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.370798 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.371164 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.371171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.371339 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.371483 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.371665 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260926 04:12:07.372380 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.361029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.0.213.254:42379)
0926 04:12:07.361122 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.361126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.361312 (+ 186us) server_negotiation.cc:407] Connection header received
0926 04:12:07.362157 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.362159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.362171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.362202 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.363359 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.364142 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.365337 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.365533 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.365660 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.368346 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.368348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.368349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.368358 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.370035 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.370511 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.370512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.370512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.370571 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.370892 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.370893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.370894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.371065 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.371149 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.372156 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.372244 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 04:12:07.372574 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.362111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.0.213.252:33387)
0926 04:12:07.362297 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.362302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.362336 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:12:07.362389 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.362391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.362424 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.362512 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.363581 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.364625 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.365600 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.365944 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.365995 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.368786 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.368794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.368794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.368801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.370723 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.371269 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.371272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.371273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.371332 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.371555 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.371556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.371557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.371744 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.371962 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.372394 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.372477 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260926 04:12:07.372874 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:07.381192 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.371796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33890)
0926 04:12:07.371849 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.371861 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.371895 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.372971 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.372973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.373140 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.373187 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.373190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.373975 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.373976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.374558 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.374560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.375001 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.375002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.375059 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.375522 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.375528 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.377062 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.379627 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.379628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.379628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.379885 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.380431 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.380433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.380433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.380511 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.380845 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.380846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.380895 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.380963 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.381071 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 04:12:07.381232 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.371969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.0.213.252:33387)
0926 04:12:07.372823 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.372825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.372833 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:12:07.372852 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.372853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.372865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.372944 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.373266 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.373849 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.374643 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.374926 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.375010 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.377159 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.377161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.377161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.377169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.379516 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.379985 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.379987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.379987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.380110 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.380600 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.380601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.380602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.380764 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.380841 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.381008 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.381091 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":123,"threads_started":1}
I20260926 04:12:07.382009 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.373390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.213.254:42379)
0926 04:12:07.373456 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.373458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.373465 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:12:07.373484 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.373484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.373493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.373549 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.374149 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.374812 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.375788 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.376062 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.376137 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.379026 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.379028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.379028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.379036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.380621 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.381053 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.381055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.381055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.381111 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.381325 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.381326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.381326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.381443 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.381602 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.381846 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.381922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 04:12:07.382949 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466: Bootstrap starting.
I20260926 04:12:07.383088 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.373447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.0.213.253:33459)
0926 04:12:07.373486 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.373495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.373503 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:12:07.373527 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.373527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.373538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.373586 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.373931 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.374662 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.375560 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.375780 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.375861 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.379114 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.379116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.379117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.379126 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.381099 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.381769 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.381771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.381771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.381836 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.382045 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.382047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.382047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.382177 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.382223 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.382941 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.383024 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:12:07.383404 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.373151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:33459 (local address 127.0.0.1:49684)
0926 04:12:07.373185 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.373195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.373232 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.373633 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.373634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.373780 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.373841 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.373843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.374725 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.374726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.375443 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.375446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.375871 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.375872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.375944 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.377198 (+ 1254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.377202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.378987 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.381189 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.381190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.381190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.381678 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.381910 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.381911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.381911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.381970 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.382766 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.382767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.382816 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.383164 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.383281 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 04:12:07.383796 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:07.383780 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:07.384176 23119 log.cc:824] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:07.384977 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466: No bootstrap required, opened a new log
I20260926 04:12:07.387092 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.387269 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4717b254a5d412f9683bef7491a2466, State: Initialized, Role: FOLLOWER
I20260926 04:12:07.387365 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.387758 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:07.387720 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.389592 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.392376 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.384121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.0.213.252:33387)
0926 04:12:07.384310 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.384312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.384318 (+ 6us) server_negotiation.cc:407] Connection header received
0926 04:12:07.384395 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.384396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.384406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.384461 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.384736 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.385444 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.386351 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.386631 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.386702 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.389250 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.389252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.389252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.389259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.391113 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.391463 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.391464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.391465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.391522 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.391694 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.391694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.391695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.391893 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.391944 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.392177 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.392276 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260926 04:12:07.392633 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.384038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33896)
0926 04:12:07.384155 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.384165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.384206 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.384465 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.384466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.384602 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.384651 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.384653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.385533 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.385534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.386264 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.386266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.386703 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.386704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.386753 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.387179 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.387184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.389151 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.391196 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.391196 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.391197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.391390 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.391586 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.391586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.391587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.391634 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.391991 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.391992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.392033 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.392451 (+ 418us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.392553 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 04:12:07.392954 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.383181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.0.213.253:33459)
0926 04:12:07.383236 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.383238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.383293 (+ 55us) server_negotiation.cc:407] Connection header received
0926 04:12:07.383381 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.383381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.383390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.383420 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.384297 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.385031 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.385861 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.386081 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.386169 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.389388 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.389389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.389390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.389397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.391298 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.391710 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.391712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.391712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.391774 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.391968 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.391969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.391970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.392122 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.392264 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.392778 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.392861 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 04:12:07.390717 23051 master_runner.cc:186] Error getting master registration for 127.0.213.253:33459: OK, instance_id { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" instance_seqno: 1790395927182660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:12:07.393778 23051 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:12:07.394443 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: Bootstrap starting.
W20260926 04:12:07.394637 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:12:07.394757 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:07.395223 23269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:07.395473 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:07.395556 23269 log.cc:824] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:07.396541 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: No bootstrap required, opened a new log
I20260926 04:12:07.396518 23093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d4717b254a5d412f9683bef7491a2466"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35078
I20260926 04:12:07.396665 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 04:12:07.396718 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:07.397405 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.397702 23093 leader_election.cc:288] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [CANDIDATE]: Term 1 election: Requested vote from peers 601f6f726c9a436a8032e1b316415a32 (127.0.213.253:33459), a3bb183f4b7a42a0a92ca818a1f42ffc (127.0.213.252:33387)
I20260926 04:12:07.399290 23269 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.399457 23269 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bb183f4b7a42a0a92ca818a1f42ffc, State: Initialized, Role: FOLLOWER
I20260926 04:12:07.399564 23269 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.399865 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.399966 23269 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:07.400018 23243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4717b254a5d412f9683bef7491a2466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc"
I20260926 04:12:07.399960 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.400151 23243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:07.401050 23243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4717b254a5d412f9683bef7491a2466 in term 1.
I20260926 04:12:07.401302 23069 leader_election.cc:302] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3bb183f4b7a42a0a92ca818a1f42ffc, d4717b254a5d412f9683bef7491a2466; no voters:
I20260926 04:12:07.401451 23276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:07.401628 23276 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [term 1 LEADER]: Becoming Leader. State: Replica: d4717b254a5d412f9683bef7491a2466, State: Running, Role: LEADER
I20260926 04:12:07.401743 23276 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.402261 23309 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4717b254a5d412f9683bef7491a2466. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.402381 23309 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: This master's current role is: LEADER
W20260926 04:12:07.402163 23308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:12:07.403365 23308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:07.403506 23312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:07.403707 23312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:07.404976 23201 master_runner.cc:186] Error getting master registration for 127.0.213.253:33459: OK, instance_id { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" instance_seqno: 1790395927182660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:12:07.407286 23201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:12:07.409488 23243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [term 1 FOLLOWER]: Refusing update from remote peer d4717b254a5d412f9683bef7491a2466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:12:07.409837 23309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:12:07.412026 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4717b254a5d412f9683bef7491a2466. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.412179 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.418334 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.399241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.0.213.252:33387)
0926 04:12:07.399281 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.399283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.408484 (+ 9201us) server_negotiation.cc:407] Connection header received
0926 04:12:07.408510 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.408511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.408521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.408571 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.409338 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.410165 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.411499 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.411808 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.411878 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.414397 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.414399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.414399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.414407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.416330 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.416983 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.416985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.417003 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.417065 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.417460 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.417461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.417462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.417607 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.417694 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.418123 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.418209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:12:07.419361 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } attempt: 3
I20260926 04:12:07.422832 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.399045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.0.213.253:33459)
0926 04:12:07.399096 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.399098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.408261 (+ 9163us) server_negotiation.cc:407] Connection header received
0926 04:12:07.408361 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.408362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.408377 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.408444 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.412588 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.413362 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.417041 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.417363 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.417420 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.418966 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.418968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.418968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.418976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.421630 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.422011 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.422012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.422012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.422070 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.422269 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.422270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.422270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.422400 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.422478 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.422662 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.422734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 04:12:07.425087 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.399039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.213.254:42379)
0926 04:12:07.399085 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.399088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.408259 (+ 9171us) server_negotiation.cc:407] Connection header received
0926 04:12:07.408380 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.408382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.408394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.408428 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.411705 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.412536 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.413973 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.414301 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.414363 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.421751 (+ 7388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.421753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.421753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.421762 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.423623 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.424101 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.424102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.424103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.424166 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.424411 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.424412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.424413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.424550 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.424639 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.424868 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.424965 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:12:07.425269 23309 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.425387 23309 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: This master's current role is: LEADER
Sep 26 04:12:07 dist-test-slave-xf24 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
I20260926 04:12:07.425915 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.426010 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.426126 23314 mvcc.cc:205] Tried to move back new op lower bound from 7333461718666383360 to 7333461718638526464. Current Snapshot: MvccSnapshot[applied={T|T < 7333461718666383360}]
I20260926 04:12:07.426328 23313 mvcc.cc:205] Tried to move back new op lower bound from 7333461718666383360 to 7333461718638526464. Current Snapshot: MvccSnapshot[applied={T|T < 7333461718666383360}]
I20260926 04:12:07.427135 23312 catalog_manager.cc:1347] Generated new cluster ID: d7a3bdafdc5c4ce2afb8bb48d70d6d7d
I20260926 04:12:07.427184 23312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:12:07.430938 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.419824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:33387 (local address 127.0.0.1:33914)
0926 04:12:07.419875 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:07.419886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:07.419924 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:07.420115 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:07.420115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:07.420277 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:07.420321 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.420322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.422158 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.422159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.422976 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:07.422978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.423847 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.423849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:07.423905 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.424359 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:07.424364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:07.426478 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:07.429721 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.429722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.429723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.429926 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.430178 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:07.430179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:07.430179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:07.430239 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:07.430637 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:07.430638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:07.430694 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 04:12:07.430742 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 04:12:07.430839 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 04:12:07.431427 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:07.419888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.0.213.252:33387)
0926 04:12:07.419939 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.419941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.419949 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:12:07.419984 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.419984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.419994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.420040 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.420635 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.422043 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.423396 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:07.423766 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:07.423834 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:07.426567 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:07.426568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:07.426568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:07.426591 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:07.429614 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.430012 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.430013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.430013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.430083 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:07.430310 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:07.430311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:07.430311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:07.430545 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:07.431137 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:07.431179 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 04:12:07.431295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 04:12:07.432937 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: Bootstrap starting.
I20260926 04:12:07.433622 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:07.433889 23194 log.cc:824] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:07.444962 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: No bootstrap required, opened a new log
I20260926 04:12:07.446971 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.447142 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f6f726c9a436a8032e1b316415a32, State: Initialized, Role: FOLLOWER
I20260926 04:12:07.447234 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } }
I20260926 04:12:07.447615 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:07.448163 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4717b254a5d412f9683bef7491a2466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "601f6f726c9a436a8032e1b316415a32"
I20260926 04:12:07.448271 23168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:07.449162 23168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4717b254a5d412f9683bef7491a2466 in term 1.
I20260926 04:12:07.450392 23167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [term 1 FOLLOWER]: Refusing update from remote peer d4717b254a5d412f9683bef7491a2466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:12:07.450664 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 04:12:07.453310 23334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:12:07.453369 23334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:07.453984 23322 mvcc.cc:205] Tried to move back new op lower bound from 7333461718666383360 to 7333461718638526464. Current Snapshot: MvccSnapshot[applied={T|T < 7333461718666383360}]
I20260926 04:12:07.456770 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.456877 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.456931 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4717b254a5d412f9683bef7491a2466. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.456987 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.457024 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.457072 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:07.458357 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4717b254a5d412f9683bef7491a2466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4717b254a5d412f9683bef7491a2466" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 42379 } } peers { permanent_uuid: "601f6f726c9a436a8032e1b316415a32" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 33459 } } peers { permanent_uuid: "a3bb183f4b7a42a0a92ca818a1f42ffc" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 33387 } } }
I20260926 04:12:07.458470 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:07.460139 23312 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:07.460916 23312 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:07.473960 23312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4717b254a5d412f9683bef7491a2466: Generated new TSK 0
I20260926 04:12:07.474300 23312 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:12:07.505602 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.503936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.0.213.253:33459)
0926 04:12:07.503982 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.503984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.504924 (+ 940us) server_negotiation.cc:407] Connection header received
0926 04:12:07.505034 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.505036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.505048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.505080 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.505527 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49710: BlockingRecv error: recv got EOF from 127.0.0.1:49710 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 04:12:07.506531 23121 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.504351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.0.213.254:42379)
0926 04:12:07.504391 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.504393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.506023 (+ 1630us) server_negotiation.cc:407] Connection header received
0926 04:12:07.506122 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.506123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.506133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.506162 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.506467 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35094: BlockingRecv error: recv got EOF from 127.0.0.1:35094 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 04:12:07.507464 23274 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.504634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.0.213.252:33387)
0926 04:12:07.504675 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.504676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.506859 (+ 2183us) server_negotiation.cc:407] Connection header received
0926 04:12:07.507037 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.507038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.507048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.507079 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.507406 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33916: BlockingRecv error: recv got EOF from 127.0.0.1:33916 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:12:07.510004 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.509252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.213.253:33459)
0926 04:12:07.509289 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.509290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.509484 (+ 194us) server_negotiation.cc:407] Connection header received
0926 04:12:07.509588 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.509589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.509608 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.509638 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.509957 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49718: BlockingRecv error: recv got EOF from 127.0.0.1:49718 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 04:12:07.513177 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.512411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.0.213.253:33459)
0926 04:12:07.512444 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.512445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.512680 (+ 235us) server_negotiation.cc:407] Connection header received
0926 04:12:07.512772 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.512773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.512783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.512811 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.513128 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49730: BlockingRecv error: recv got EOF from 127.0.0.1:49730 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 04:12:07.517176 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.516441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.0.213.253:33459)
0926 04:12:07.516474 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.516476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.516677 (+ 201us) server_negotiation.cc:407] Connection header received
0926 04:12:07.516771 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.516772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.516783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.516812 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.517128 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49744: BlockingRecv error: recv got EOF from 127.0.0.1:49744 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:12:07.523182 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.522389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.0.213.253:33459)
0926 04:12:07.522428 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.522430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.522669 (+ 239us) server_negotiation.cc:407] Connection header received
0926 04:12:07.522763 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.522764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.522774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.522803 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.523130 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49758: BlockingRecv error: recv got EOF from 127.0.0.1:49758 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:12:07.537040 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.536010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.213.253:33459)
0926 04:12:07.536075 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.536078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.536350 (+ 272us) server_negotiation.cc:407] Connection header received
0926 04:12:07.536477 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.536478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.536494 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.536528 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.536962 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49760: BlockingRecv error: recv got EOF from 127.0.0.1:49760 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 04:12:07.555482 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.554495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.0.213.253:33459)
0926 04:12:07.554540 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.554543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.554844 (+ 301us) server_negotiation.cc:407] Connection header received
0926 04:12:07.554956 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.554957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.554971 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.555004 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.555416 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49772: BlockingRecv error: recv got EOF from 127.0.0.1:49772 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:12:07.589952 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.588905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.0.213.253:33459)
0926 04:12:07.588953 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.588955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.589290 (+ 335us) server_negotiation.cc:407] Connection header received
0926 04:12:07.589417 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.589418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.589430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.589462 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.589893 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49782: BlockingRecv error: recv got EOF from 127.0.0.1:49782 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:12:07.656211 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.655454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.0.213.253:33459)
0926 04:12:07.655545 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.655547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.655597 (+ 50us) server_negotiation.cc:407] Connection header received
0926 04:12:07.655643 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.655644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.655655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.655709 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.656144 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49792: BlockingRecv error: recv got EOF from 127.0.0.1:49792 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 04:12:07.786106 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:07.785365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.0.213.253:33459)
0926 04:12:07.785460 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:07.785463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:07.785491 (+ 28us) server_negotiation.cc:407] Connection header received
0926 04:12:07.785590 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:07.785591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:07.785602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:07.785658 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:07.786052 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49804: BlockingRecv error: recv got EOF from 127.0.0.1:49804 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 04:12:08.044272 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:08.043522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.213.253:33459)
0926 04:12:08.043616 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:08.043618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:08.043649 (+ 31us) server_negotiation.cc:407] Connection header received
0926 04:12:08.043679 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:08.043680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:08.043692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:08.043768 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:08.044184 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49818: BlockingRecv error: recv got EOF from 127.0.0.1:49818 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 04:12:08.403855 23308 catalog_manager.cc:1259] Loaded cluster ID: d7a3bdafdc5c4ce2afb8bb48d70d6d7d
I20260926 04:12:08.403982 23308 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: loading cluster ID for follower catalog manager: success
I20260926 04:12:08.404865 23308 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: acquiring CA information for follower catalog manager: success
I20260926 04:12:08.405198 23308 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3bb183f4b7a42a0a92ca818a1f42ffc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:12:08.453876 23334 catalog_manager.cc:1259] Loaded cluster ID: d7a3bdafdc5c4ce2afb8bb48d70d6d7d
I20260926 04:12:08.453999 23334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: loading cluster ID for follower catalog manager: success
I20260926 04:12:08.454869 23334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: acquiring CA information for follower catalog manager: success
I20260926 04:12:08.455200 23334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 601f6f726c9a436a8032e1b316415a32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:12:08.560549 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:08.557639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.0.213.253:33459)
0926 04:12:08.557752 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:08.557754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:08.557820 (+ 66us) server_negotiation.cc:407] Connection header received
0926 04:12:08.557903 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:08.557904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:08.557988 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:08.558082 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:08.558242 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.558759 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.559639 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.559864 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.559942 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:08.560164 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:08.560295 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:08.560388 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:08.560411 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 04:12:08.560437 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260926 04:12:08.564837 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:08.562501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.213.252:33387)
0926 04:12:08.562606 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:08.562608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:08.562640 (+ 32us) server_negotiation.cc:407] Connection header received
0926 04:12:08.562683 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:08.562683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:08.562718 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:08.562780 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:08.562866 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.563336 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.564124 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.564331 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.564385 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:08.564549 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:08.564652 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:08.564706 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:08.564754 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 04:12:08.564771 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260926 04:12:08.569695 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:08.566643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.0.213.253:33459)
0926 04:12:08.566686 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:08.566689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:08.566702 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:12:08.566810 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:08.566811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:08.566824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:08.566881 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:08.567120 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.567896 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.568920 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.569223 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.569291 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:08.569434 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:08.569461 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:08.569524 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:08.569588 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 04:12:08.569609 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:12:08.569700 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:08.567084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.0.213.252:33387)
0926 04:12:08.567122 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:08.567123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:08.567186 (+ 63us) server_negotiation.cc:407] Connection header received
0926 04:12:08.567267 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:08.567268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:08.567277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:08.567332 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:08.567431 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.567970 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.568865 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.569079 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.569156 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:08.569439 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:08.569471 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:08.569514 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:08.569598 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 04:12:08.569620 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:12:08.570490 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:08.566779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.0.213.254:42379)
0926 04:12:08.566878 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:08.566880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:08.567110 (+ 230us) server_negotiation.cc:407] Connection header received
0926 04:12:08.567219 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:08.567220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:08.567272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:08.567346 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:08.567544 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.568334 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.569354 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:08.569677 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:08.569747 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:08.569979 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:08.570090 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:08.570131 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:08.570374 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 04:12:08.570404 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260926 04:12:08.572482 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 23051
I20260926 04:12:08.577905 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 23126
I20260926 04:12:08.583611 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 23201
2026-09-26T04:12:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2887 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:12:08 dist-test-slave-xf24 krb5kdc[23418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:12:08 dist-test-slave-xf24 krb5kdc[23418](info): set up 2 sockets
Sep 26 04:12:08 dist-test-slave-xf24 krb5kdc[23418](info): commencing operation
krb5kdc: starting...
W20260926 04:12:09.628418 855 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/test-user.keytab.
Sep 26 04:12:09 dist-test-slave-xf24 krb5kdc[23418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
Entry for principal kudu/127.0.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab.
I20260926 04:12:09.756027 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46263
--webserver_interface=127.0.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46263,127.0.213.253:34685,127.0.213.252:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:12:09.844014 23431 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:12:09.844269 23431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:09.844288 23431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:09.844308 23431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:09.845436 23431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:09.845474 23431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:09.845489 23431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:09.845502 23431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:09.845513 23431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:09.846982 23431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46263,127.0.213.253:34685,127.0.213.252:36431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.254.keytab
--principal=kudu/127.0.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.254:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:09.847208 23431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:09.847383 23431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:09.852588 23436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:09.852602 23439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:09.852599 23437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:09.852820 23431 server_base.cc:1034] running on GCE node
I20260926 04:12:09.853114 23431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:09.853477 23431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:09.854657 23431 hybrid_clock.cc:648] HybridClock initialized: now 1790395929854651 us; error 40 us; skew 500 ppm
Sep 26 04:12:09 dist-test-slave-xf24 krb5kdc[23418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:09.860657 23431 init.cc:377] Logged in from keytab as kudu/127.0.213.254@KRBTEST.COM (short username kudu)
I20260926 04:12:09.861176 23431 webserver.cc:460] Webserver started at http://127.0.213.254:41359/ using document root <none> and password file <none>
I20260926 04:12:09.861425 23431 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:09.861505 23431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:09.861660 23431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:09.862776 23431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data/instance:
uuid: "54335ad91a144c43bc731f498f7852c5"
format_stamp: "Formatted at 2026-09-26 04:12:09 on dist-test-slave-xf24"
server_key: "4901ff70fbda0172f4beccd875c24ec9"
server_key_iv: "7b65e45f2d060ed956df43012dd887d4"
server_key_version: "encryptionkey@0"
I20260926 04:12:09.863160 23431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal/instance:
uuid: "54335ad91a144c43bc731f498f7852c5"
format_stamp: "Formatted at 2026-09-26 04:12:09 on dist-test-slave-xf24"
server_key: "4901ff70fbda0172f4beccd875c24ec9"
server_key_iv: "7b65e45f2d060ed956df43012dd887d4"
server_key_version: "encryptionkey@0"
I20260926 04:12:09.864697 23431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:12:09.865628 23446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:09.865865 23431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:12:09.865978 23431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal
uuid: "54335ad91a144c43bc731f498f7852c5"
format_stamp: "Formatted at 2026-09-26 04:12:09 on dist-test-slave-xf24"
server_key: "4901ff70fbda0172f4beccd875c24ec9"
server_key_iv: "7b65e45f2d060ed956df43012dd887d4"
server_key_version: "encryptionkey@0"
I20260926 04:12:09.866083 23431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:09.881984 23431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:09.884159 23431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:09.884325 23431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:09.888429 23431 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.254:46263
I20260926 04:12:09.888479 23498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.254:46263 every 8 connection(s)
I20260926 04:12:09.888863 23431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/data/info.pb
I20260926 04:12:09.889839 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:09.891542 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 23431
I20260926 04:12:09.891636 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-0/wal/instance
I20260926 04:12:09.891531 23499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:09.891844 855 external_mini_cluster.cc:1428] Setting key 632bd55ad1f02b58de94e6f25fe864e3
Sep 26 04:12:09 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.253@KRBTEST.COM" created.
I20260926 04:12:09.902472 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:09.892414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46263 (local address 127.0.0.1:35502)
0926 04:12:09.892663 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:09.892699 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:09.893179 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:09.893492 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:09.893495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:09.893787 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:09.893938 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:09.893942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:09.894961 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:09.894962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:09.895737 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:09.895740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:09.896350 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:09.896351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:09.896418 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:09.897062 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:09.897071 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:09.898807 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:09.900855 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:09.900862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:09.900864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:09.901065 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:09.901343 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:09.901344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:09.901345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:09.901455 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:09.901850 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:09.901855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:09.901921 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 04:12:09.902029 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 04:12:09.902270 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":95,"thread_start_us":121,"threads_started":1}
I20260926 04:12:09.902469 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:09.892491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.0.213.254:46263)
0926 04:12:09.892659 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:09.892670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:09.892802 (+ 132us) server_negotiation.cc:407] Connection header received
0926 04:12:09.893366 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:09.893374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:09.893390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:09.893493 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:09.894047 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:09.894854 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:09.895842 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:09.896234 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:09.896373 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:09.898921 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:09.898928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:09.898928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:09.898936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:09.900753 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:09.901163 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:09.901167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:09.901167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:09.901271 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:09.901536 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:09.901536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:09.901537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:09.901771 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:09.901862 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:09.902063 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 04:12:09.902207 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260926 04:12:09.903192 23499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:09.903570 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:09.903448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:48980)
0926 04:12:09.903499 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:09.903545 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:12:09.903769 23448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
I20260926 04:12:09.904251 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:09.904152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:48996)
0926 04:12:09.904204 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:09.904228 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 04:12:09.904358 23499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:34685: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab.
Entry for principal kudu/127.0.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab.
I20260926 04:12:09.920764 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:34685
--webserver_interface=127.0.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab
--principal=kudu/127.0.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46263,127.0.213.253:34685,127.0.213.252:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:12:09.953613 23499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } attempt: 1
I20260926 04:12:09.954281 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:09.954064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49006)
0926 04:12:09.954194 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:09.954247 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 04:12:09.954749 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:09.954612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49018)
0926 04:12:09.954676 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:09.954715 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 04:12:09.954885 23499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:34685: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
W20260926 04:12:10.003978 23506 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:12:10.004240 23506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:10.004257 23506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:10.004279 23506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:10.005674 23506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:10.005739 23506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:10.005754 23506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:10.005769 23506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:10.005781 23506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:10.007828 23506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46263,127.0.213.253:34685,127.0.213.252:36431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.253.keytab
--principal=kudu/127.0.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.253:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:10.008186 23506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:10.008486 23506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:12:10.013996 23514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:10.014077 23506 server_base.cc:1034] running on GCE node
W20260926 04:12:10.014019 23511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:10.014055 23512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:10.014436 23506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:10.014776 23506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:10.016016 23506 hybrid_clock.cc:648] HybridClock initialized: now 1790395930015999 us; error 33 us; skew 500 ppm
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:10.018978 23506 init.cc:377] Logged in from keytab as kudu/127.0.213.253@KRBTEST.COM (short username kudu)
I20260926 04:12:10.019544 23506 webserver.cc:460] Webserver started at http://127.0.213.253:37847/ using document root <none> and password file <none>
I20260926 04:12:10.019804 23506 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:10.019878 23506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:10.020135 23506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:10.021313 23506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data/instance:
uuid: "bd247c9de67d47c38de2ce0ebeb732b4"
format_stamp: "Formatted at 2026-09-26 04:12:10 on dist-test-slave-xf24"
server_key: "1256dca4c8a07dca9d3137118117f450"
server_key_iv: "b23aa6749e2d5b6bedbaa20dc81a0a05"
server_key_version: "encryptionkey@0"
I20260926 04:12:10.021703 23506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal/instance:
uuid: "bd247c9de67d47c38de2ce0ebeb732b4"
format_stamp: "Formatted at 2026-09-26 04:12:10 on dist-test-slave-xf24"
server_key: "1256dca4c8a07dca9d3137118117f450"
server_key_iv: "b23aa6749e2d5b6bedbaa20dc81a0a05"
server_key_version: "encryptionkey@0"
I20260926 04:12:10.023033 23499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } attempt: 2
I20260926 04:12:10.023185 23506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:12:10.023541 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.023388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49034)
0926 04:12:10.023458 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.023499 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 04:12:10.024026 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.023850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49040)
0926 04:12:10.023919 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.023982 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 04:12:10.024152 23499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.253:34685: Network error: Client connection negotiation failed: client connection to 127.0.213.253:34685: connect: Connection refused (error 111)
I20260926 04:12:10.024246 23521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:10.024474 23506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:12:10.024552 23506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal
uuid: "bd247c9de67d47c38de2ce0ebeb732b4"
format_stamp: "Formatted at 2026-09-26 04:12:10 on dist-test-slave-xf24"
server_key: "1256dca4c8a07dca9d3137118117f450"
server_key_iv: "b23aa6749e2d5b6bedbaa20dc81a0a05"
server_key_version: "encryptionkey@0"
I20260926 04:12:10.024639 23506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:10.032601 23506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:10.035116 23506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:10.035300 23506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:10.040144 23506 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.253:34685
I20260926 04:12:10.040213 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.253:34685 every 8 connection(s)
I20260926 04:12:10.040581 23506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/data/info.pb
I20260926 04:12:10.041618 23574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:10.043210 23574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:10.046029 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 23506
I20260926 04:12:10.046169 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-1/wal/instance
I20260926 04:12:10.046389 855 external_mini_cluster.cc:1428] Setting key 387cf68ee28a57e0b71b1d3bab3dde7a
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:12:10.054522 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.043955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.213.254:46263)
0926 04:12:10.044104 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.044106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.044250 (+ 144us) server_negotiation.cc:407] Connection header received
0926 04:12:10.044742 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.044743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.044754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.044824 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.045506 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.045998 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.047587 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.047806 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.047920 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.050734 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.050735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.050736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.050742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.052942 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.053439 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.053440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.053440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.053513 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.053760 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.053762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.053762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.053916 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.053989 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.054275 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.054370 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:12:10.054529 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.043894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46263 (local address 127.0.0.1:35512)
0926 04:12:10.044091 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.044137 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.044614 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.044910 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.044914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.045192 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.045398 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.045402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.046101 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.046103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.047467 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.047471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.047948 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.047950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.048166 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.048773 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.048783 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.050600 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.053053 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.053060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.053062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.053344 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.053587 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.053588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.053588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.053692 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.053983 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.053988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.054144 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.054230 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.054357 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.213.252@KRBTEST.COM; defaulting to no policy
I20260926 04:12:10.055248 23574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.213.252@KRBTEST.COM" created.
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:10.062872 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.055549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49042)
0926 04:12:10.055605 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.055618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.055671 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.055925 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.055926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.056146 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.056215 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.056217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.057114 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.057115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.057684 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.057686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.058263 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.058263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.058330 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.058784 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.058789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.059977 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.061787 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.061788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.061789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.061956 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.062181 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.062182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.062182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.062236 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.062543 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.062543 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.062582 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.062649 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.062755 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 04:12:10.063047 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.055673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.0.213.253:34685)
0926 04:12:10.055802 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.055805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.055820 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:12:10.055836 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.055837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.055847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.055897 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.056296 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.056985 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.057804 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.058176 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.058255 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.060112 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.060120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.060120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.060127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.061693 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.062060 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.062067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.062068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.062114 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.062316 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.062317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.062317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.062462 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.062552 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.062688 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.062826 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260926 04:12:10.063678 23574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:10.064109 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.063898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60552)
0926 04:12:10.063952 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.064040 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 04:12:10.064241 23523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
I20260926 04:12:10.064692 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.064599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60562)
0926 04:12:10.064647 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.064671 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:12:10.064803 23574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:36431: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab.
Entry for principal kudu/127.0.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab.
I20260926 04:12:10.075956 855 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
/tmp/dist-test-taskrg1Sr7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:36431
--webserver_interface=127.0.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab
--principal=kudu/127.0.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.213.212:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.213.254:46263,127.0.213.253:34685,127.0.213.252:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:12:10.114068 23574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } attempt: 1
I20260926 04:12:10.114723 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.114518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60574)
0926 04:12:10.114642 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.114689 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 04:12:10.115216 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.115071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60580)
0926 04:12:10.115148 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.115190 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 04:12:10.115288 23499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } attempt: 3
W20260926 04:12:10.115477 23574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:36431: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:10.122993 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.115701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.0.213.253:34685)
0926 04:12:10.115757 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.115759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.115771 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:12:10.115890 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.115891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.115911 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.115989 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.116383 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.116882 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.117645 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.117874 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.117957 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.120211 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.120213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.120213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.120222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.121820 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.122197 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.122198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.122199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.122258 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.122460 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.122461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.122461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.122583 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.122660 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.122808 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.122891 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:12:10.122997 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.115651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49050)
0926 04:12:10.115709 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.115734 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.115785 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.116005 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.116006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.116224 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.116300 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.116301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.116982 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.116983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.117544 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.117545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.117955 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.117956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.118040 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.118616 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.118621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.120012 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.121937 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.121938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.121939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.122119 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.122337 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.122338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.122338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.122393 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.122651 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.122652 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.122706 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.122784 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.122891 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 04:12:10.123430 23499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:12:10.123836 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.123701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60590)
0926 04:12:10.123774 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.123809 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 04:12:10.124236 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.124137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60606)
0926 04:12:10.124187 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.124212 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 04:12:10.124336 23499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:36431: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
I20260926 04:12:10.155515 23499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } attempt: 1
I20260926 04:12:10.156121 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.155900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60608)
0926 04:12:10.155993 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.156030 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 04:12:10.156512 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.156378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60610)
0926 04:12:10.156451 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.156480 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 04:12:10.156606 23499 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:36431: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
W20260926 04:12:10.161183 23581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:12:10.161414 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:12:10.161433 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:12:10.161456 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:12:10.162753 23581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:12:10.162819 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:12:10.162835 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:12:10.162848 23581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:12:10.162863 23581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:12:10.164647 23581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.213.212:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.213.254:46263,127.0.213.253:34685,127.0.213.252:36431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/krb5kdc/kudu_127.0.213.252.keytab
--principal=kudu/127.0.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.213.252:36431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:12:10.164952 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:12:10.165217 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:12:10.170833 23581 server_base.cc:1034] running on GCE node
W20260926 04:12:10.170811 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:10.170801 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:12:10.170812 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:12:10.171278 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:12:10.171633 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:12:10.172830 23581 hybrid_clock.cc:648] HybridClock initialized: now 1790395930172773 us; error 75 us; skew 500 ppm
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:12:10.175554 23581 init.cc:377] Logged in from keytab as kudu/127.0.213.252@KRBTEST.COM (short username kudu)
I20260926 04:12:10.176156 23581 webserver.cc:460] Webserver started at http://127.0.213.252:40693/ using document root <none> and password file <none>
I20260926 04:12:10.176398 23581 fs_manager.cc:359] Metadata directory not provided
I20260926 04:12:10.176476 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:12:10.176625 23581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:12:10.177748 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data/instance:
uuid: "63f4c521f0354dc8aa826a27670e8768"
format_stamp: "Formatted at 2026-09-26 04:12:10 on dist-test-slave-xf24"
server_key: "f71b0ea76f914a302f97bbfc4a60f247"
server_key_iv: "818baf76cae3f73103a3e5fae9fe56a4"
server_key_version: "encryptionkey@0"
I20260926 04:12:10.178128 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal/instance:
uuid: "63f4c521f0354dc8aa826a27670e8768"
format_stamp: "Formatted at 2026-09-26 04:12:10 on dist-test-slave-xf24"
server_key: "f71b0ea76f914a302f97bbfc4a60f247"
server_key_iv: "818baf76cae3f73103a3e5fae9fe56a4"
server_key_version: "encryptionkey@0"
I20260926 04:12:10.179630 23581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:12:10.180512 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:12:10.180752 23581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:12:10.180876 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data,/tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal
uuid: "63f4c521f0354dc8aa826a27670e8768"
format_stamp: "Formatted at 2026-09-26 04:12:10 on dist-test-slave-xf24"
server_key: "f71b0ea76f914a302f97bbfc4a60f247"
server_key_iv: "818baf76cae3f73103a3e5fae9fe56a4"
server_key_version: "encryptionkey@0"
I20260926 04:12:10.180994 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:12:10.183650 23574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } attempt: 2
I20260926 04:12:10.184206 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.184033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60616)
0926 04:12:10.184126 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.184172 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 04:12:10.184660 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.184523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60630)
0926 04:12:10.184584 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.184621 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:12:10.184785 23574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.213.252:36431: Network error: Client connection negotiation failed: client connection to 127.0.213.252:36431: connect: Connection refused (error 111)
I20260926 04:12:10.205470 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:12:10.207592 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:12:10.207772 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:12:10.212041 23581 rpc_server.cc:307] RPC server started. Bound to: 127.0.213.252:36431
I20260926 04:12:10.212131 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.213.252:36431 every 8 connection(s)
I20260926 04:12:10.212527 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/data/info.pb
I20260926 04:12:10.213543 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:12:10.215093 23649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:12:10.221557 855 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu as pid 23581
I20260926 04:12:10.221675 855 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrg1Sr7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395654118158-855-0/minicluster-data/master-2/wal/instance
I20260926 04:12:10.221920 855 external_mini_cluster.cc:1428] Setting key dd31248d45bb601a05bd91d6604ad86d
I20260926 04:12:10.224985 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.215834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.0.213.254:46263)
0926 04:12:10.215934 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.215936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.216127 (+ 191us) server_negotiation.cc:407] Connection header received
0926 04:12:10.216628 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.216630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.216641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.216704 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.217355 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.217884 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.218859 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.219083 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.219144 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.221497 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.221499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.221500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.221540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.223343 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.223835 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.223836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.223836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.223907 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.224242 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.224243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.224244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.224370 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.224453 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.224772 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.224882 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 04:12:10.225178 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.215763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.254:46263 (local address 127.0.0.1:35520)
0926 04:12:10.215941 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.215987 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.216531 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.216777 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.216780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.217067 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.217267 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.217271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.217966 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.217966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.218780 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.218782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.219172 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.219173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.219278 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.219785 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.219794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.221383 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.223475 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.223483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.223486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.223740 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.223984 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.223985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.223986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.224169 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.224463 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.224470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.224625 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.224736 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.224906 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260926 04:12:10.226123 23649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:10.231735 23499 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } attempt: 2
I20260926 04:12:10.233019 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.222422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.0.213.252:36431)
0926 04:12:10.222660 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.222666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.222922 (+ 256us) server_negotiation.cc:407] Connection header received
0926 04:12:10.222985 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.222988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.223005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.223081 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.223742 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.224609 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.225692 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.226164 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.226250 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.229233 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.229244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.229244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.229249 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.231029 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.231509 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.231514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.231514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.231599 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.231847 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.231848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.231849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.232338 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.232536 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.232582 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.232668 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":144,"threads_started":1}
I20260926 04:12:10.234099 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.226546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.213.253:34685)
0926 04:12:10.226642 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.226644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.226655 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:12:10.226758 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.226759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.226770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.226861 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.227186 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.227675 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.228544 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.228840 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.228905 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.231040 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.231042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.231043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.231057 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.232815 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.233263 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.233264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.233264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.233343 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.233543 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.233544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.233544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.233686 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.233757 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.233923 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.234007 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 04:12:10.234130 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.226441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.253:34685 (local address 127.0.0.1:49064)
0926 04:12:10.226607 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.226617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.226666 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.226873 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.226874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.227033 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.227097 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.227099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.227740 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.227742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.228432 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.228435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.228903 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.228904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.228976 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.229417 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.229422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.230938 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.232894 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.232895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.232895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.233177 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.233413 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.233414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.233414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.233471 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.233751 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.233752 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.233799 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.233885 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.233999 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 04:12:10.234496 23649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.254@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.252@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
I20260926 04:12:10.239977 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.232097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.0.213.252:36431)
0926 04:12:10.232295 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.232297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.232305 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:12:10.232324 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.232325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.232336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.232743 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.232791 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.233553 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.234620 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.234973 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.235311 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.237122 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.237123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.237124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.237131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.238838 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.239202 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.239203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.239203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.239264 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.239469 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.239470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.239470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.239591 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.239669 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.239799 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.239870 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":102,"threads_started":1}
I20260926 04:12:10.239977 23501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.232015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60640)
0926 04:12:10.232129 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.232141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.232182 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.232434 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.232435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.232609 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.232668 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.232670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.233702 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.233703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.234529 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.234533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.235072 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.235073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.235136 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.235594 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.235599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.237061 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.238940 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.238941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.238941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.239127 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.239338 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.239339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.239339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.239405 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.239662 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.239663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.239707 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.239775 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.239870 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.254@KRBTEST.COM
I20260926 04:12:10.241708 23499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5: Bootstrap starting.
I20260926 04:12:10.242375 23499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:10.242753 23499 log.cc:824] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:10.243602 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.234805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60646)
0926 04:12:10.234850 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.234860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.234900 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.235108 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.235109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.235252 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.235288 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.235289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.236203 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.236204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.236974 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.236977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.237989 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.237990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.238054 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.238392 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.238396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.240474 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.242478 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.242479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.242480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.242701 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.242871 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.242871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.242872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.242925 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.243235 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.243236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.243277 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.243350 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.243459 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 04:12:10.243680 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.234910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.0.213.252:36431)
0926 04:12:10.235007 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.235009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.235016 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:12:10.235040 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.235041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.235053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.235103 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.235368 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.236127 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.237060 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.237910 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.237978 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.240554 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.240555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.240555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.240562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.242383 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.242757 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.242757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.242758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.242794 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.242998 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.242999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.243000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.243183 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.243244 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.243379 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.243520 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":80}
I20260926 04:12:10.243970 23499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5: No bootstrap required, opened a new log
I20260926 04:12:10.244517 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.234962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.0.213.254:46263)
0926 04:12:10.235014 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.235016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.235416 (+ 400us) server_negotiation.cc:407] Connection header received
0926 04:12:10.235507 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.235508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.235518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.235547 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.236211 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.236658 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.237627 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.237929 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.237971 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.241018 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.241019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.241020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.241026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.242546 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.243059 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.243060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.243060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.243109 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.243304 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.243305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.243305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.243444 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.243583 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.244320 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.244406 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:12:10.245143 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: Bootstrap starting.
I20260926 04:12:10.245864 23649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:10.246227 23649 log.cc:824] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:10.246232 23499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.246361 23499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54335ad91a144c43bc731f498f7852c5, State: Initialized, Role: FOLLOWER
I20260926 04:12:10.246459 23499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.246706 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.246779 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:10.246794 23499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:10.248867 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: No bootstrap required, opened a new log
W20260926 04:12:10.249161 23668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:12:10.249267 23668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:10.250473 23649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.250625 23649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f4c521f0354dc8aa826a27670e8768, State: Initialized, Role: FOLLOWER
I20260926 04:12:10.250716 23649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.250993 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.251065 23649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:10.252529 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.213.253@KRBTEST.COM
I20260926 04:12:10.249626 23431 master_runner.cc:186] Error getting master registration for 127.0.213.253:34685: OK, instance_id { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" instance_seqno: 1790395930038597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:12:10.254279 23431 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 04:12:10.253249 23681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:12:10.254810 23681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:10.255424 23581 master_runner.cc:186] Error getting master registration for 127.0.213.253:34685: OK, instance_id { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" instance_seqno: 1790395930038597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:12:10.257828 23581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:12:10.258172 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.245249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.213.253:34685)
0926 04:12:10.245343 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.245346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.245353 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:12:10.245377 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.245377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.245395 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.245464 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.245989 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.246726 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.248254 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.248584 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.248663 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.254532 (+ 5869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.254534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.254534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.254543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.256543 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.256966 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.256968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.256968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.257042 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.257243 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.257244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.257245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.257385 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.257538 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.257982 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.258076 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 04:12:10.259683 855 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:12:10.260769 23473 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "54335ad91a144c43bc731f498f7852c5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35534
I20260926 04:12:10.260921 23473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 04:12:10.260970 23473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:10.262039 23473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.262302 23473 leader_election.cc:288] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [CANDIDATE]: Term 1 election: Requested vote from peers bd247c9de67d47c38de2ce0ebeb732b4 (127.0.213.253:34685), 63f4c521f0354dc8aa826a27670e8768 (127.0.213.252:36431)
I20260926 04:12:10.262846 23623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54335ad91a144c43bc731f498f7852c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "63f4c521f0354dc8aa826a27670e8768"
I20260926 04:12:10.262984 23623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:10.263890 23623 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54335ad91a144c43bc731f498f7852c5 in term 1.
I20260926 04:12:10.264159 23448 leader_election.cc:302] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54335ad91a144c43bc731f498f7852c5, 63f4c521f0354dc8aa826a27670e8768; no voters:
I20260926 04:12:10.264334 23657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:12:10.264544 23657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [term 1 LEADER]: Becoming Leader. State: Replica: 54335ad91a144c43bc731f498f7852c5, State: Running, Role: LEADER
I20260926 04:12:10.264645 23657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.265095 23687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54335ad91a144c43bc731f498f7852c5. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.265311 23687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:10.265661 23689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:12:10.265877 23689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:12:10.267923 23623 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [term 1 FOLLOWER]: Refusing update from remote peer 54335ad91a144c43bc731f498f7852c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:12:10.268399 23687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:12:10.270155 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54335ad91a144c43bc731f498f7852c5. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.270254 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:10.271121 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.271219 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:10.271370 23691 mvcc.cc:205] Tried to move back new op lower bound from 7333461730373939200 to 7333461730364776448. Current Snapshot: MvccSnapshot[applied={T|T < 7333461730373939200}]
I20260926 04:12:10.271478 23690 mvcc.cc:205] Tried to move back new op lower bound from 7333461730373939200 to 7333461730364776448. Current Snapshot: MvccSnapshot[applied={T|T < 7333461730373939200}]
I20260926 04:12:10.271590 23670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.271669 23670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:10.271973 23689 catalog_manager.cc:1347] Generated new cluster ID: f4f1af181ee344a5aef5fc91ff2c1786
I20260926 04:12:10.272022 23689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:12:10.275928 23574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } attempt: 3
Sep 26 04:12:10 dist-test-slave-xf24 krb5kdc[23418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.213.253@KRBTEST.COM for kudu/127.0.213.252@KRBTEST.COM
I20260926 04:12:10.285605 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.276304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.213.252:36431 (local address 127.0.0.1:60658)
0926 04:12:10.276429 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:12:10.276440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:12:10.276491 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:12:10.276741 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:12:10.276742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:12:10.276931 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:12:10.276975 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.276977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.277988 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.277990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.278814 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:12:10.278816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.279381 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.279382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:12:10.279439 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.280576 (+ 1137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:12:10.280581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:12:10.282116 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:12:10.284396 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.284397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.284398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.284626 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.284942 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:12:10.284943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:12:10.284943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:12:10.285006 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:12:10.285278 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:12:10.285279 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:12:10.285325 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 04:12:10.285386 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 04:12:10.285499 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 04:12:10.285953 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.276567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.213.252:36431)
0926 04:12:10.276606 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.276608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.276615 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:12:10.276638 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.276639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.276651 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.276697 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.277074 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.277885 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.278937 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.279260 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.279323 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.282215 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.282216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.282216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.282224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.284246 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.284791 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.284792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.284793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.284868 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.285070 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.285071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.285071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.285198 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.285253 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.285778 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.285866 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:12:10.286854 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.272357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.213.253:34685)
0926 04:12:10.272408 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.272411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.272965 (+ 554us) server_negotiation.cc:407] Connection header received
0926 04:12:10.273138 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.273152 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.273164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.273196 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.274632 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.275427 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.277483 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.278271 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.278321 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.281289 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.281290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.281291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.281300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.284395 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.285097 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.285099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.285099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.285163 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.285490 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.285491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.285491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.285616 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.285702 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.286687 (+ 985us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.286766 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:12:10.287561 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: Bootstrap starting.
I20260926 04:12:10.288429 23574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: Neither blocks nor log segments found. Creating new log.
I20260926 04:12:10.288781 23574 log.cc:824] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: Log is configured to *not* fsync() on all Append() calls
I20260926 04:12:10.289352 23689 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:12:10.290189 23689 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:12:10.291262 23653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.272605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.213.252:36431)
0926 04:12:10.272655 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.272657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.281851 (+ 9194us) server_negotiation.cc:407] Connection header received
0926 04:12:10.281942 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.281943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.281954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.281984 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.282597 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.283270 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.286680 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.286984 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.287034 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.288485 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.288487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.288487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.288494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.290135 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.290517 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.290518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.290519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.290568 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.290738 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.290738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.290739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.290848 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.290921 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.291104 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.291175 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 04:12:10.300235 23689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5: Generated new TSK 0
I20260926 04:12:10.300797 23689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:12:10.301048 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: No bootstrap required, opened a new log
I20260926 04:12:10.303273 23574 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.303449 23574 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd247c9de67d47c38de2ce0ebeb732b4, State: Initialized, Role: FOLLOWER
I20260926 04:12:10.303550 23574 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } }
I20260926 04:12:10.303910 23574 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:12:10.303946 23548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54335ad91a144c43bc731f498f7852c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd247c9de67d47c38de2ce0ebeb732b4"
I20260926 04:12:10.304507 23548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:12:10.305507 23548 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54335ad91a144c43bc731f498f7852c5 in term 1.
I20260926 04:12:10.305707 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:10.276147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.0.213.254:46263)
0926 04:12:10.276360 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.276362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.280708 (+ 4346us) server_negotiation.cc:407] Connection header received
0926 04:12:10.280821 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.280823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.280836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.280867 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.281753 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.282538 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.285958 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:10.286259 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:10.286303 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:10.292176 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:12:10.292177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:12:10.292178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:12:10.292186 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:12:10.293773 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.295785 (+ 2012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.295787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.295788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.295873 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:12:10.300296 (+ 4423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:12:10.300298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:12:10.300298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:12:10.300465 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:12:10.300583 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:10.305459 (+ 4876us) server_negotiation.cc:299] Negotiation successful
0926 04:12:10.305604 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260926 04:12:10.306248 23547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [term 1 FOLLOWER]: Refusing update from remote peer 54335ad91a144c43bc731f498f7852c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:12:10.306800 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.306948 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:10.307331 23657 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:12:10.309624 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54335ad91a144c43bc731f498f7852c5. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.309777 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:12:10.309852 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.309947 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:12:10.310153 23710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:12:10.310222 23710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:12:10.312589 23711 mvcc.cc:205] Tried to move back new op lower bound from 7333461730503749632 to 7333461730364776448. Current Snapshot: MvccSnapshot[applied={T|T < 7333461730373939200}]
W20260926 04:12:10.333256 23502 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.331450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.0.213.254:46263)
0926 04:12:10.331498 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.331500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.332495 (+ 995us) server_negotiation.cc:407] Connection header received
0926 04:12:10.332612 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.332613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.332626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.332658 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.333167 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35564: BlockingRecv error: recv got EOF from 127.0.0.1:35564 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 04:12:10.334033 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.331871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.0.213.253:34685)
0926 04:12:10.331914 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.331916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.333515 (+ 1599us) server_negotiation.cc:407] Connection header received
0926 04:12:10.333616 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.333617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.333628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.333658 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.333969 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49080: BlockingRecv error: recv got EOF from 127.0.0.1:49080 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 04:12:10.334808 23653 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.332232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.213.252:36431)
0926 04:12:10.332287 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.332290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.334291 (+ 2001us) server_negotiation.cc:407] Connection header received
0926 04:12:10.334391 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.334392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.334403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.334433 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.334747 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60666: BlockingRecv error: recv got EOF from 127.0.0.1:60666 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 04:12:10.340917 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.336775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.0.213.253:34685)
0926 04:12:10.336820 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.336822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.340278 (+ 3456us) server_negotiation.cc:407] Connection header received
0926 04:12:10.340409 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.340410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.340434 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.340465 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.340853 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49096: BlockingRecv error: recv got EOF from 127.0.0.1:49096 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:12:10.344266 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.343359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.0.213.253:34685)
0926 04:12:10.343400 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.343402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.343674 (+ 272us) server_negotiation.cc:407] Connection header received
0926 04:12:10.343774 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.343775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.343786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.343816 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.344205 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49102: BlockingRecv error: recv got EOF from 127.0.0.1:49102 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 04:12:10.348562 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.347587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.0.213.253:34685)
0926 04:12:10.347627 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.347630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.347909 (+ 279us) server_negotiation.cc:407] Connection header received
0926 04:12:10.348013 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.348014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.348025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.348079 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.348501 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49110: BlockingRecv error: recv got EOF from 127.0.0.1:49110 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:12:10.357358 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.356422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.0.213.253:34685)
0926 04:12:10.356467 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.356469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.356755 (+ 286us) server_negotiation.cc:407] Connection header received
0926 04:12:10.356859 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.356860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.356872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.356902 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.357296 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49112: BlockingRecv error: recv got EOF from 127.0.0.1:49112 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 04:12:10.367653 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.366713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.0.213.253:34685)
0926 04:12:10.366756 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.366758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.367033 (+ 275us) server_negotiation.cc:407] Connection header received
0926 04:12:10.367156 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.367157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.367169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.367200 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.367583 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49124: BlockingRecv error: recv got EOF from 127.0.0.1:49124 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 04:12:10.386049 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.385141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.0.213.253:34685)
0926 04:12:10.385186 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.385188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.385455 (+ 267us) server_negotiation.cc:407] Connection header received
0926 04:12:10.385561 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.385562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.385574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.385604 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.385989 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49134: BlockingRecv error: recv got EOF from 127.0.0.1:49134 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:12:10.420519 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.419545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.0.213.253:34685)
0926 04:12:10.419593 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.419595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.419875 (+ 280us) server_negotiation.cc:407] Connection header received
0926 04:12:10.419982 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.419983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.419995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.420026 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.420446 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49140: BlockingRecv error: recv got EOF from 127.0.0.1:49140 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 04:12:10.486672 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.485981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.0.213.253:34685)
0926 04:12:10.486057 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.486060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.486078 (+ 18us) server_negotiation.cc:407] Connection header received
0926 04:12:10.486164 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.486165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.486175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.486228 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.486624 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49142: BlockingRecv error: recv got EOF from 127.0.0.1:49142 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 04:12:10.616837 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.616088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.213.253:34685)
0926 04:12:10.616188 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.616190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.616260 (+ 70us) server_negotiation.cc:407] Connection header received
0926 04:12:10.616306 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.616307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.616318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.616387 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.616766 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49154: BlockingRecv error: recv got EOF from 127.0.0.1:49154 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 04:12:10.875277 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:12:10.874442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.0.213.253:34685)
0926 04:12:10.874628 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:10.874631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:10.874684 (+ 53us) server_negotiation.cc:407] Connection header received
0926 04:12:10.874715 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:10.874716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:10.874729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:10.874800 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:12:10.875182 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49160: BlockingRecv error: recv got EOF from 127.0.0.1:49160 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
I20260926 04:12:10.915771 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54335ad91a144c43bc731f498f7852c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54335ad91a144c43bc731f498f7852c5" member_type: VOTER last_known_addr { host: "127.0.213.254" port: 46263 } } peers { permanent_uuid: "bd247c9de67d47c38de2ce0ebeb732b4" member_type: VOTER last_known_addr { host: "127.0.213.253" port: 34685 } } peers { permanent_uuid: "63f4c521f0354dc8aa826a27670e8768" member_type: VOTER last_known_addr { host: "127.0.213.252" port: 36431 } } }
I20260926 04:12:10.915934 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54335ad91a144c43bc731f498f7852c5 [sys.catalog]: This master's current role is: LEADER
I20260926 04:12:11.255316 23681 catalog_manager.cc:1259] Loaded cluster ID: f4f1af181ee344a5aef5fc91ff2c1786
I20260926 04:12:11.255391 23681 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: loading cluster ID for follower catalog manager: success
I20260926 04:12:11.256309 23681 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: acquiring CA information for follower catalog manager: success
I20260926 04:12:11.256619 23681 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63f4c521f0354dc8aa826a27670e8768: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:12:11.310678 23710 catalog_manager.cc:1259] Loaded cluster ID: f4f1af181ee344a5aef5fc91ff2c1786
I20260926 04:12:11.310786 23710 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: loading cluster ID for follower catalog manager: success
I20260926 04:12:11.311704 23710 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: acquiring CA information for follower catalog manager: success
I20260926 04:12:11.312040 23710 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd247c9de67d47c38de2ce0ebeb732b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:12:11.391495 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:11.388629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.0.213.253:34685)
0926 04:12:11.388777 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:11.388778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:11.388790 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:12:11.388858 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:11.388858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:11.388909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:11.389010 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:11.389257 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.389838 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.390658 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.390871 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.390950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:11.391129 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:11.391252 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:11.391333 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:11.391379 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 04:12:11.391398 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260926 04:12:11.396212 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:11.393512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.0.213.252:36431)
0926 04:12:11.393642 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:11.393644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:11.393652 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:12:11.393687 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:11.393687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:11.393736 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:11.393806 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:11.393911 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.394423 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.395375 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.395582 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.395646 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:11.395808 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:11.395918 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:11.396011 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:11.396045 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 04:12:11.396134 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260926 04:12:11.401379 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:11.398097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.0.213.253:34685)
0926 04:12:11.398142 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:11.398144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:11.398292 (+ 148us) server_negotiation.cc:407] Connection header received
0926 04:12:11.398367 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:11.398368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:11.398383 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:11.398441 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:11.398617 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.399439 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.400519 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.400801 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.400860 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:11.401066 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:11.401104 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:11.401150 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:11.401250 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 04:12:11.401282 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:12:11.401933 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:11.398491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.213.252:36431)
0926 04:12:11.398538 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:11.398540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:11.398547 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:12:11.398574 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:11.398575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:11.398583 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:11.398630 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:11.398824 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.399616 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.400675 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.400903 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.400983 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:11.401146 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:11.401181 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:11.401234 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:11.401797 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 04:12:11.401823 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 04:12:11.402242 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:12:11.398186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.0.213.254:46263)
0926 04:12:11.398399 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 04:12:11.398402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:12:11.398410 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:12:11.398468 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:12:11.398469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:12:11.398525 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:12:11.398612 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:12:11.398839 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.399366 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.400898 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:12:11.401264 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:12:11.401333 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:12:11.401519 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:12:11.401659 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:12:11.401913 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 04:12:11.402009 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 04:12:11.402151 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":51,"threads_started":1}
I20260926 04:12:11.404328 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 23431
I20260926 04:12:11.410233 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 23506
I20260926 04:12:11.415234 855 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrg1Sr7/build/release/bin/kudu with pid 23581
2026-09-26T04:12:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2831 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5719 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277303 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST